Bayesian Dimension Reduction for Regressions in High Dimensions

Abstract

We aim to find a dimension reduction technique in high-dimensional (p ≫ n) and correlated data settings to explain and predict relevant quantities. A common approach in these high-dimensional settings is to use some frequen- tist dimension reduction method first and then fit a predictive model on the reduced space. However, many reduction methods have poor performance or are computationally infeasible for high dimensions. Additionally, such two-step approaches fail to keep track of the uncer- tainty in the reduction for the predictions. We propose several adaptations and extensions to an existing Bayesian method called Targeted Random Pro- jection (TARP, Mukhopadhyay and Dunson 2020). It combines a probabilis- tic variable screening step with a random projection step to obtain a sparse set of reduced variables. These are used to fit a simple linear model for pre- diction. This procedure of screening, projection and prediction is repeated several times to explore different reductions. The obtained set of smaller predictive models is then aggregated by model averaging to get overall pre- dictions. New insights into this method are pointed out including suggested adap- tions in the targeting step and possible robust extensions to handle response outliers. In extensive simulations and a real data application we show the ad- vantages of these adaptions and extensions over the original method and other competitors.