Sparse data-driven random projection in regression for high-dimensional data

Abstract

We examine the linear regression problem in a challenging high-dimensional setting with correlated predictors to explain and predict relevant quantities, with explicitly allowing the regression coefficient to vary from sparse to dense. Most classical high-dimensional regression estimators require some degree of sparsity. We discuss the more recent concept of random projection as computationally fast dimension reduction tool, and propose a new random projection matrix tailored to the linear regression problem with a theoretical bound on the gain in expected prediction error over conventional random projections. Around this new random projection, we built the Sparse Projected Averaged Regression (SPAR) method combining probabilistic variable screening steps with the random projection steps to obtain an ensemble of small linear models with a thresholding parameter to obtain a higher degree of sparsity. In extensive simulations and a real data application we compare prediction and variable ranking performance to various competitors.