Abstract:This paper presents Sparse Gradient Descent as a solution for variable selection in convex piecewise linear regression where the model is given as $\mathrm{max}\langle a_j^\star, x \rangle + b_j^\star$ for $j = 1,\dots,k$ where $x \in \mathbb R^d$ is the covariate vector. Here, $\{a_j^\star\}_{j=1}^k$ and $\{b_j^\star\}_{j=1}^k$ denote the ground-truth weight vectors and intercepts. A non-asymptotic local convergence analysis is provided for Sp-GD under sub-Gaussian noise when the covariate distribution satisfies sub-Gaussianity and anti-concentration property. When the model order and parameters are fixed, Sp-GD provides an $\epsilon$-accurate estimate given $\mathcal{O}(\max(\epsilon^{-2}\sigma_z^2,1)s\log(d/s))$ observations where $\sigma_z^2$ denotes the noise variance. This also implies the exact parameter recovery by Sp-GD from $\mathcal{O}(s\log(d/s))$ noise-free observations. Since optimizing the squared loss for sparse max-affine is non-convex, an initialization scheme is proposed to provide a suitable initial estimate within the basin of attraction for Sp-GD, i.e. sufficiently accurate to invoke the convergence guarantees. The initialization scheme uses sparse principal component analysis to estimate the subspace spanned by $\{ a_j^\star\}_{j=1}^k$ then applies an $r$-covering search to estimate the model parameters. A non-asymptotic analysis is presented for this initialization scheme when the covariates and noise samples follow Gaussian distributions. When the model order and parameters are fixed, this initialization scheme provides an $\epsilon$-accurate estimate given $\mathcal{O}(\epsilon^{-2}\max(\sigma_z^4,\sigma_z^2,1)s^2\log^4(d))$ observations. Numerical Monte Carlo results corroborate theoretical findings for Sp-GD and the initialization scheme.

Learning Model-Based Sparsity via Projected Gradient Descent

Convergence of Projected Subgradient Method with Sparse or Low-Rank Constraints

Greedy Sparsity-Constrained Optimization

Smoothing Proximal Gradient Method for General Structured Sparse Learning

Tree-Projected Gradient Descent for Estimating Gradient-Sparse Parameters on Graphs

Sparse Learning for Large-Scale and High-Dimensional Data: A Randomized Convex-Concave Optimization Approach

Nonconvex Sparse Logistic Regression Via Proximal Gradient Descent.

A Unified Dynamic Approach to Sparse Model Selection

Slowly Varying Regression under Sparsity

High-dimensional Grouped-regression using Bayesian Sparse Projection-posterior

Bayesian High-dimensional Linear Regression with Sparse Projection-posterior

Linear Convergence of Stochastic Iterative Greedy Algorithms With Sparse Constraints

Variable Selection in Convex Piecewise Linear Regression

A Semismooth Newton Algorithm for High-Dimensional Nonconvex Sparse Learning

Quadratic Approximation Greedy Pursuit for Cardinality-Constrained Sparse Learning

Estimation of off-the grid sparse spikes with over-parametrized projected gradient descent: theory and application

An Alternating Proximal Splitting Method With Global Convergence For Nonconvex Structured Sparsity Optimization

Gradient-based sparse principal component analysis with extensions to online learning

Smoothing the Edges: Smooth Optimization for Sparse Regularization using Hadamard Overparametrization

Learning Sparsity-Promoting Regularizers Using Bilevel Optimization

Multi-stage convex relaxation for learning with sparse regularization