Abstract:This paper presents Sparse Gradient Descent as a solution for variable selection in convex piecewise linear regression where the model is given as $\mathrm{max}\langle a_j^\star, x \rangle + b_j^\star$ for $j = 1,\dots,k$ where $x \in \mathbb R^d$ is the covariate vector. Here, $\{a_j^\star\}_{j=1}^k$ and $\{b_j^\star\}_{j=1}^k$ denote the ground-truth weight vectors and intercepts. A non-asymptotic local convergence analysis is provided for Sp-GD under sub-Gaussian noise when the covariate distribution satisfies sub-Gaussianity and anti-concentration property. When the model order and parameters are fixed, Sp-GD provides an $\epsilon$-accurate estimate given $\mathcal{O}(\max(\epsilon^{-2}\sigma_z^2,1)s\log(d/s))$ observations where $\sigma_z^2$ denotes the noise variance. This also implies the exact parameter recovery by Sp-GD from $\mathcal{O}(s\log(d/s))$ noise-free observations. Since optimizing the squared loss for sparse max-affine is non-convex, an initialization scheme is proposed to provide a suitable initial estimate within the basin of attraction for Sp-GD, i.e. sufficiently accurate to invoke the convergence guarantees. The initialization scheme uses sparse principal component analysis to estimate the subspace spanned by $\{ a_j^\star\}_{j=1}^k$ then applies an $r$-covering search to estimate the model parameters. A non-asymptotic analysis is presented for this initialization scheme when the covariates and noise samples follow Gaussian distributions. When the model order and parameters are fixed, this initialization scheme provides an $\epsilon$-accurate estimate given $\mathcal{O}(\epsilon^{-2}\max(\sigma_z^4,\sigma_z^2,1)s^2\log^4(d))$ observations. Numerical Monte Carlo results corroborate theoretical findings for Sp-GD and the initialization scheme.

Variable selection for classification with derivative-induced regularization

Faithful Variable Screening for High-Dimensional Convex Regression

Variable Selection for Generalized Varying Coefficient Partially Linear Models with Diverging Number of Parameters

Regularization and variable selection for infinite variance autoregressive models

EFFICIENT KERNEL-BASED VARIABLE SELECTION WITH SPARSISTENCY

Robust Variable Selection and Estimation Based on Kernel Modal Regression

Bayesian variable selection in linear regression models with instrumental variables

Gradient-induced Model-free Variable Selection Based on Composite Quantile Regression in Reproducing Kernel Hilbert Space

Variable Selection in Convex Piecewise Linear Regression

Variable Selection in a Class of Single-Index Models

Variable Selection for High Dimensional Partially Linear Varying Coefficient Errors-in-variables Models

A Variance Minimization Criterion to Feature Selection Using Laplacian Regularization

Estimation and Variable Selection for Generalized Functional Partially Varying Coefficient Hybrid Models

Gradient Learning Approach For Variable Selection In Credit Scoring

GRADIENT-INDUCED MODEL-FREE VARIABLE SELECTION WITH COMPOSITE QUANTILE REGRESSION

Towards a Unified Framework for Uncertainty-aware Nonlinear Variable Selection with Theoretical Guarantees

Variable selection in model-based clustering and discriminant analysis with a regularization approach

Structured variable selection in support vector machines

Variable Selection and Minimax Prediction in High-dimensional Functional Linear Model

Variable Selection in High-Dimensional Quantile Varying Coefficient Models

Variable selection for partially linear single-index varying-coefficient model