Abstract:Suppose that we first apply the Lasso to a design matrix, and then update one of its columns. In general, the signs of the Lasso coefficients may change, and there is no closed-form expression for updating the Lasso solution exactly. In this work, we propose an approximate formula for updating a debiased Lasso coefficient. We provide general nonasymptotic error bounds in terms of the norms and correlations of a given design matrix's columns, and then prove asymptotic convergence results for the case of a random design matrix with i.i.d.\ sub-Gaussian row vectors and i.i.d.\ Gaussian noise. Notably, the approximate formula is asymptotically correct for most coordinates in the proportional growth regime, under the mild assumption that each row of the design matrix is sub-Gaussian with a covariance matrix having a bounded condition number. Our proof only requires certain concentration and anti-concentration properties to control various error terms and the number of sign changes. In contrast, rigorously establishing distributional limit properties (e.g.\ Gaussian limits for the debiased Lasso) under similarly general assumptions has been considered open problem in the universality theory. As applications, we show that the approximate formula allows us to reduce the computation complexity of variable selection algorithms that require solving multiple Lasso problems, such as the conditional randomization test and a variant of the knockoff filter.

Error bounds of constant gain least-mean-squares algorithms

A Bound on Mean-Square Estimation Error with Background Parameter Mismatch

On the Bounds of the Modelling Errors of Black-Box MIMO Transfer Function Estimates

On the Mean-Square Performance of the Constrained LMS Algorithm

Performance Analysis of $l_0$ Norm Constraint Least Mean Square Algorithm

Regularized Least-Mean-Square Algorithms

On the convergence analysis of the transform domain normalized LMS and related M-estimate algorithms

Modifier Formula on Mean Square Convergence of LMS Algorithm

Mean-square error constrained approach to robust stochastic iterative learning control

Performance Analysis of L-0-Lms with Gaussian Input Signal

Variable p norm constrained LMS algorithm based on gradient of root relative deviation.pdf

$L_{0}$ Norm Constraint LMS Algorithm for Sparse System Identification

Stability of a Generalized Debiased Lasso with Applications to Resampling-Based Variable Selection

Convergence Behavior of Gauss-Newton's Method and Extensions of the Smale Point Estimate Theory.

Adaptive least-squares finite element methods: Guaranteed upper bounds and convergence in L 2 norm of the dual variable

l0 Norm Constraint LMS Algorithm for Sparse System Identification.

A Direction Optimization Least Mean Square Algorithm

Strong convergence rate of the least median absolute estimator in linear regression models

New Error Analysis for Lasso

Computation of lower bounds for the induced L2 norm of LPV systems

R-Linear Convergence of Limited Memory Steepest Descent