Abstract:We present the framework of slowly varying regression under sparsity, allowing sparse regression models to exhibit slow and sparse variations. The problem of parameter estimation is formulated as a mixed-integer optimization problem. We demonstrate that it can be precisely reformulated as a binary convex optimization problem through a novel relaxation technique. This relaxation involves a new equality on Moore-Penrose inverses, convexifying the non-convex objective function while matching the original objective on all feasible binary points. This enables us to efficiently solve the problem to provable optimality using a cutting plane-type algorithm. We develop a highly optimized implementation of this algorithm, substantially improving upon the asymptotic computational complexity of a straightforward implementation. Additionally, we propose a fast heuristic method that guarantees a feasible solution and, as empirically illustrated, produces high-quality warm-start solutions for the binary optimization problem. To tune the framework's hyperparameters, we suggest a practical procedure relying on binary search that, under certain assumptions, is guaranteed to recover the true model parameters. On both synthetic and real-world datasets, we demonstrate that the resulting algorithm outperforms competing formulations in comparable times across various metrics, including estimation accuracy, predictive power, and computational time. The algorithm is highly scalable, allowing us to train models with thousands of parameters. Our implementation is available open-source at <a class="link-external link-https" href="https://github.com/vvdigalakis/SSVRegression.git" rel="external noopener nofollow">this https URL</a>.

Sparse Unit-Sum Regression

On The Sparse Bayesian Learning Of Linear Models

Conditional Sparse Linear Regression

On the Conditions of Sparse Parameter Estimation via Log-Sum Penalty Regularization.

Slowly Varying Regression under Sparsity

On Regularized Sparse Logistic Regression

Sparse quantile regression

A Sparse-Group Lasso

Sparse Deep Learning Models with the $\ell_1$ Regularization

Sparse Index Tracking: Simultaneous Asset Selection and Capital Allocation via $\ell_0$-Constrained Portfolio

Sparse Portfolio Selection via Quasi-Norm Regularization

Analyzing Weighted $\ell_1$ Minimization for Sparse Recovery with Nonuniform Sparse Models\footnote{The results of this paper were presented in part at the International Symposium on Information Theory, ISIT 2009}

Jointly Sparse Global SIMPLS Regression

Sparse Linear Regression with Constraints: A Flexible Entropy-based Framework

On Sparse Estimation for Semiparametric Linear Transformation Models

Iterative Reweighted Framework Based Algorithms for Sparse Linear Regression with Generalized Elastic Net Penalty

Reduced-Space Iteratively Reweighted Second-Order Methods for Nonconvex Sparse Regularization

Nonsparse learning with latent variables

A minimax optimal approach to high-dimensional double sparse linear regression

Bayesian High-dimensional Linear Regression with Sparse Projection-posterior

Scaled sparse linear regression