Abstract:This paper studies schemes to de-bias the Lasso in a linear model $y=X\beta+\epsilon$ where the goal is to construct confidence intervals for $a_0^T\beta$ in a direction $a_0$, where $X$ has iid $N(0,\Sigma)$ rows. We show that previously analyzed propositions to de-bias the Lasso require a modification in order to enjoy efficiency in a full range of sparsity. This modification takes the form of a degrees-of-freedom adjustment that accounts for the dimension of the model selected by Lasso. Let $s_0$ be the true sparsity. If $\Sigma$ is known and the ideal score vector proportional to $X\Sigma^{-1}a_0$ is used, the unadjusted de-biasing schemes proposed previously enjoy efficiency if $s_0\lll n^{2/3}$. However, if $s_0\ggg n^{2/3}$, the unadjusted schemes cannot be efficient in certain $a_0$: then it is necessary to modify existing procedures by a degrees-of-freedom adjustment. This modification grants asymptotic efficiency for any $a_0$ when $s_0/p\to 0$ and $s_0\log(p/s_0)/n \to 0$. If $\Sigma$ is unknown, efficiency is granted for general $a_0$ when $$\frac{s_0\log p}{n}+\min\Big\{\frac{s_\Omega\log p}{n},\frac{\|\Sigma^{-1}a_0\|_1\sqrt{\log p}}{\|\Sigma^{-1/2}a_0\|_2 \sqrt n}\Big\}+\frac{\min(s_\Omega,s_0)\log p}{\sqrt n}\to0$$ where $s_\Omega=\|\Sigma^{-1}a_0\|_0$, provided that the de-biased estimate is modified with the degrees-of-freedom adjustment. The dependence in $s_0,s_\Omega$ and $\|\Sigma^{-1}a_0\|_1$ is optimal. Our estimated score vector provides a novel methodology to handle dense $a_0$. Our analysis shows that the degrees-of-freedom adjustment is not needed when the initial bias in direction $a_0$ is small, which is granted under stringent conditions on $\Sigma^{-1}$. The main proof argument is an interpolation path similar to that typically used to derive Slepian's lemma. It yields a new $\ell_\infty$ error bound for the Lasso which is of independent interest.

The Dual and Degrees of Freedom of Linearly Constrained Generalized Lasso

Adaptive Lasso in high-dimensional settings

A Fast Solution to the Lasso Problem with Equality Constraints

Stability of a Generalized Debiased Lasso with Applications to Resampling-Based Variable Selection

Sparse Estimation Via ℓ_q Optimization Method in High-Dimensional Linear Regression

New Dual Formulations In Constrained Integer Programming

Dual Lasso Selector

A descent algorithm for constrained LAD-Lasso estimation with applications in portfolio selection

Efficient Lasso Training from a Geometrical Perspective.

De-Biasing The Lasso With Degrees-of-Freedom Adjustment

Statistical inference for high dimensional regression via Constrained Lasso

Constraint Qualifications for Optimality Conditions and Total Lagrange Dualities in Convex Infinite Programming

Mind the duality gap: safer rules for the Lasso

A Canonical Dual Approach for Solving Linearly Constrained Quadratic Programs

Near-Optimal Solutions of Constrained Learning Problems

The Adaptive Lasso and Its Oracle Properties

DOUBLY DEBIASED LASSO: HIGH-DIMENSIONAL INFERENCE UNDER HIDDEN CONFOUNDING.

A Proximal Point Algorithm For Log-Determinant Optimization With Group Lasso Regularization

Simultaneous Inference for High-Dimensional Linear Models

New Error Analysis for Lasso

Geometric Approach and Closed Exact Formulae for the Lasso