Abstract:Sparse linear regression (SLR) is a well-studied problem in statistics where one is given a design matrix $X\in\mathbb{R}^{m\times n}$ and a response vector $y=X\theta^*+w$ for a $k$-sparse vector $\theta^*$ (that is, $\|\theta^*\|_0\leq k$) and small, arbitrary noise $w$, and the goal is to find a $k$-sparse $\widehat{\theta} \in \mathbb{R}^n$ that minimizes the mean squared prediction error $\frac{1}{m}\|X\widehat{\theta}-X\theta^*\|^2_2$. While $\ell_1$-relaxation methods such as basis pursuit, Lasso, and the Dantzig selector solve SLR when the design matrix is well-conditioned, no general algorithm is known, nor is there any formal evidence of hardness in an average-case setting with respect to all efficient algorithms. We give evidence of average-case hardness of SLR w.r.t. all efficient algorithms assuming the worst-case hardness of lattice problems. Specifically, we give an instance-by-instance reduction from a variant of the bounded distance decoding (BDD) problem on lattices to SLR, where the condition number of the lattice basis that defines the BDD instance is directly related to the restricted eigenvalue condition of the design matrix, which characterizes some of the classical statistical-computational gaps for sparse linear regression. Also, by appealing to worst-case to average-case reductions from the world of lattices, this shows hardness for a distribution of SLR instances; while the design matrices are ill-conditioned, the resulting SLR instances are in the identifiable regime. Furthermore, for well-conditioned (essentially) isotropic Gaussian design matrices, where Lasso is known to behave well in the identifiable regime, we show hardness of outputting any good solution in the unidentifiable regime where there are many solutions, assuming the worst-case hardness of standard and well-studied lattice problems.

Lattice Designs in Standard and Simple Implicit Multi-linear Regression

Introduction to Implicit Regression

Covariate-adjusted response-adaptive designs for generalized linear models

The adaptive Wynn-algorithm in generalized linear models with univariate response

Implicit Regression: Detecting Constants and Inverse Relationships with Bivariate Random Error

Sparse Linear Regression and Lattice Problems

Computational Tools for Probing Interactions in Multiple Linear Regression, Multilevel Modeling, and Latent Curve Analysis

R-optimal designs for multi-response regression models with multi-factors

L2RM: Low-Rank Linear Regression Models for High-Dimensional Matrix Responses

On the implied weights of linear regression for causal inference

D-Optimal Designs for the Mitscherlich Non-Linear Regression Function

Optimal Designs in Sparse Linear Models

On the minimax robustness against correlation and heteroscedasticity of ordinary least squares among generalized least squares estimators of regression

Indirect multivariate response linear regression

Selection of Regression Models under Linear Restrictions for Fixed and Random Designs

Fast Screening Rules for Optimal Design via Quadratic Lasso Reformulation

Simple or complex statistical models: Non-traditional regression models with intuitive interpretations

Lasso Adjustments Of Treatment Effect Estimates In Randomized Experiments

Potential weights and implicit causal designs in linear regression

Optimal Designs for Minimax-Criteria in Random Coefficient Regression Models

Stability of a Generalized Debiased Lasso with Applications to Resampling-Based Variable Selection