Abstract:This paper presents Sparse Gradient Descent as a solution for variable selection in convex piecewise linear regression where the model is given as $\mathrm{max}\langle a_j^\star, x \rangle + b_j^\star$ for $j = 1,\dots,k$ where $x \in \mathbb R^d$ is the covariate vector. Here, $\{a_j^\star\}_{j=1}^k$ and $\{b_j^\star\}_{j=1}^k$ denote the ground-truth weight vectors and intercepts. A non-asymptotic local convergence analysis is provided for Sp-GD under sub-Gaussian noise when the covariate distribution satisfies sub-Gaussianity and anti-concentration property. When the model order and parameters are fixed, Sp-GD provides an $\epsilon$-accurate estimate given $\mathcal{O}(\max(\epsilon^{-2}\sigma_z^2,1)s\log(d/s))$ observations where $\sigma_z^2$ denotes the noise variance. This also implies the exact parameter recovery by Sp-GD from $\mathcal{O}(s\log(d/s))$ noise-free observations. Since optimizing the squared loss for sparse max-affine is non-convex, an initialization scheme is proposed to provide a suitable initial estimate within the basin of attraction for Sp-GD, i.e. sufficiently accurate to invoke the convergence guarantees. The initialization scheme uses sparse principal component analysis to estimate the subspace spanned by $\{ a_j^\star\}_{j=1}^k$ then applies an $r$-covering search to estimate the model parameters. A non-asymptotic analysis is presented for this initialization scheme when the covariates and noise samples follow Gaussian distributions. When the model order and parameters are fixed, this initialization scheme provides an $\epsilon$-accurate estimate given $\mathcal{O}(\epsilon^{-2}\max(\sigma_z^4,\sigma_z^2,1)s^2\log^4(d))$ observations. Numerical Monte Carlo results corroborate theoretical findings for Sp-GD and the initialization scheme.

Technical Note-Knowledge Gradient for Selection with Covariates: Consistency and Computation

Discretization-free Knowledge Gradient Methods for Bayesian Optimization

Non-Myopic Knowledge Gradient Policy for Ranking and Selection.

Improving the Knowledge Gradient Algorithm

Sequential Sampling for Bayesian Robust Ranking and Selection

Knowledge Gradient for Multi-Objective Bayesian Optimization with Decoupled Evaluations

Optimal Learning for Sequential Decision Making for Expensive Cost Functions with Stochastic Binary Feedbacks

The Knowledge Gradient with Logistic Belief Models for Binary Classification

Gradient-Based Myopic Allocation Policy: an Efficient Sampling Procedure in a Low-Confidence Scenario.

Gradient is All You Need?

On-Line Policy Gradient Estimation with Multi-Step Sampling.

Gradient Information Matters in Policy Optimization by Back-propagating through Model

Error Analysis of Stochastic Gradient Descent Ranking.

A Sharp Analysis of Covariate Adjusted Precision Matrix Estimation Via Alternating Projected Gradient Descent

Gradient play in stochastic games: stationary points, convergence, and sample complexity

Shuffling Gradient Descent-Ascent with Variance Reduction for Nonconvex-Strongly Concave Smooth Minimax Problems

A Temporal-Difference Approach to Policy Gradient Estimation

Gradient-induced variable selection in reproducing kernel Hilbert space for survival analysis

An improvement of policy gradient estimation algorithms

Practical and Matching Gradient Variance Bounds for Black-Box Variational Bayesian Inference

Variable Selection in Convex Piecewise Linear Regression