Abstract:We present the framework of slowly varying regression under sparsity, allowing sparse regression models to exhibit slow and sparse variations. The problem of parameter estimation is formulated as a mixed-integer optimization problem. We demonstrate that it can be precisely reformulated as a binary convex optimization problem through a novel relaxation technique. This relaxation involves a new equality on Moore-Penrose inverses, convexifying the non-convex objective function while matching the original objective on all feasible binary points. This enables us to efficiently solve the problem to provable optimality using a cutting plane-type algorithm. We develop a highly optimized implementation of this algorithm, substantially improving upon the asymptotic computational complexity of a straightforward implementation. Additionally, we propose a fast heuristic method that guarantees a feasible solution and, as empirically illustrated, produces high-quality warm-start solutions for the binary optimization problem. To tune the framework's hyperparameters, we suggest a practical procedure relying on binary search that, under certain assumptions, is guaranteed to recover the true model parameters. On both synthetic and real-world datasets, we demonstrate that the resulting algorithm outperforms competing formulations in comparable times across various metrics, including estimation accuracy, predictive power, and computational time. The algorithm is highly scalable, allowing us to train models with thousands of parameters. Our implementation is available open-source at <a class="link-external link-https" href="https://github.com/vvdigalakis/SSVRegression.git" rel="external noopener nofollow">this https URL</a>.

Hierarchical sparsity priors for regression models

Leveraging Joint Sparsity in Hierarchical Bayesian Learning

Bayesian High-dimensional Linear Regression with Sparse Projection-posterior

High-dimensional Grouped-regression using Bayesian Sparse Projection-posterior

A hierarchical prior for generalized linear models based on predictions for the mean response

Dynamic sparsity on dynamic regression models

Adaptive posterior concentration rates for sparse high-dimensional linear regression with random design and unknown error variance

Sparse Bayesian Hierarchical Modeling of High-dimensional Clustering Problems

Bayesian variable selection in linear regression models with instrumental variables

Neuronized Priors for Bayesian Sparse Linear Regression

Adaptive Bayesian Predictive Inference in High-dimensional Regerssion

Tail-adaptive Bayesian shrinkage

Empirical Bayes inference in sparse high-dimensional generalized linear models

Sparsity enforcing priors in inverse problems via Normal variance mixtures: model selection, algorithms and applications

Slowly Varying Regression under Sparsity

Intuitive joint priors for variance parameters

Sparsity promoting hybrid solvers for hierarchical Bayesian inverse problems

Sparse reconstructions from few noisy data: analysis of hierarchical Bayesian models with generalized gamma hyperpriors

Computationally Efficient Sampling Methods for Sparsity Promoting Hierarchical Bayesian Models