A Unified Framework for Pattern Recovery in Penalized and Thresholded Estimation and its Geometry

Abstract

We consider the framework of penalized estimation where the penalty term is given by a real-valued polyhedral gauge, which encompasses methods such as LASSO, generalized LASSO, SLOPE, OSCAR, PACS and others. Each of these estimators is defined through an optimization problem and can uncover a different structure or “pattern” of the unknown parameter vector. We define a novel and general notion of patterns based on subdifferentials and formalize an approach to measure pattern complexity. For pattern recovery, we provide a minimal condition for a particular pattern to be detected by the procedure with positive probability, the so-called accessibility condition. Using our approach, we also introduce the stronger noiseless recovery condition. For the LASSO, it is well known that the irrepresentability condition is necessary for pattern recovery with probability larger than 1/2 and we show that the noiseless recovery plays exactly the same role in our general framework, thereby unifying and extending the irrepresentability condition to a broad class of penalized estimators. We also show that the noiseless recovery condition can be relaxed when turning to so-called thresholded penalized estimators: we prove that the necessary condition of accessibility is already sufficient for sure pattern recovery by thresholded penalized estimation provided that the noise is small enough. Throughout the article, we demonstrate how our findings can be interpreted through a geometrical lens.