Abstract:We provide a geometric approach to the lasso. We study the tangency of the level sets of the least square objective function with the polyhedral boundary sets $B(t)$ of the parameters in $\mathbb R^p$ with the $\ell_1$ norm equal to $t$. Here $t$ decreases from the value $\hat t$, which corresponds to the actual, nonconstrained minimizer of the least square objective function, denoted by $\hat\beta$. We derive closed exact formulae for the solution of the lasso under the full rank assumption. Our method does not assume iterative numerical procedures and it is, thus, computationally more efficient than the existing algorithms for solving the lasso. We also establish several important general properties of the solutions of the lasso. We prove that each lasso solution form a simple polygonal chain in $\mathbb{R}^p$ with $\hat\beta$ and the origin as the endpoints. There are no two segments of the polygonal chain that are parallel. We prove that such a polygonal chain can intersect interiors of more than one orthant in $\mathbb{R}^p$, but it cannot intersect interiors of more than $p$ orthants, which is, in general, the best possible estimate for non-normalized data. We prove that if a polygonal chain passes from the interior of one to the interior of another orthant, then it never again returns to the interior of the former. The intersection of a chain and the interior of an orthant coincides with a segment minus its end points, which belongs to a ray having $\hat\beta$ as its initial point. We illustrate the results using real data examples as well as especially crafted examples with hypothetical data. Already in $p=2$ case we show a striking difference in the maximal number of quadrants a polygonal chain of a lasso solution can intersect in the case of normalized data, which is $1$ vs. nonnormalized data, which is $2$.

The Group Lasso for Design of Experiments

A note on the group lasso and a sparse group lasso

Adaptive Lasso in high-dimensional settings

Coarse optimization for complex systems: an application of orthogonal experiments

On the design-dependent suboptimality of the Lasso

A Sparse-Group Lasso

Optimal Designs in Sparse Linear Models

Lasso Adjustments Of Treatment Effect Estimates In Randomized Experiments

Fast Screening Rules for Optimal Design via Quadratic Lasso Reformulation

Design-Based Theory for Lasso Adjustment in Randomized Block Experiments and Rerandomized Experiments

A cost-sensitive constrained Lasso

An efficient Hessian based algorithm for solving large-scale sparse group Lasso problems

A Proximal Point Algorithm For Log-Determinant Optimization With Group Lasso Regularization

Geometric Approach and Closed Exact Formulae for the Lasso

Consistent group selection in high-dimensional linear regression

Properties and Iterative Methods for the Lasso and Its Variants

Overlapping Group Lasso for High-Dimensional Generalized Linear Models

Nearly Column-Orthogonal Designs Based on Leave-One-Out Good Lattice Point Sets

Causal Inference of Social Experiments Using Orthogonal Designs

Exclusive Group Lasso for Structured Variable Selection