Abstract:Given a high-dimensional covariate matrix and a response vector, ridge-regularized sparse linear regression selects a subset of features that explains the relationship between covariates and the response in an interpretable manner. To select the sparsity and robustness of linear regressors, techniques like k-fold cross-validation are commonly used for hyperparameter tuning. However, cross-validation substantially increases the computational cost of sparse regression as it requires solving many mixed-integer optimization problems (MIOs). Additionally, validation metrics often serve as noisy estimators of test set errors, with different hyperparameter combinations leading to models with different noise levels. Therefore, optimizing over these metrics is vulnerable to out-of-sample disappointment, especially in underdetermined settings. To improve upon this state of affairs, we make two key contributions. First, motivated by the generalization theory literature, we propose selecting hyperparameters that minimize a weighted sum of a cross-validation metric and a model's output stability, thus reducing the risk of poor out-of-sample performance. Second, we leverage ideas from the mixed-integer optimization literature to obtain computationally tractable relaxations of k-fold cross-validation metrics and the output stability of regressors, facilitating hyperparameter selection after solving fewer MIOs. These relaxations result in an efficient cyclic coordinate descent scheme, achieving lower validation errors than via traditional methods such as grid search. On synthetic datasets, our confidence adjustment procedure improves out-of-sample performance by 2%-5% compared to minimizing the k-fold error alone. On 13 real-world datasets, our confidence adjustment procedure reduces test set error by 2%, on average.

Branch and Bound to Assess Stability of Regression Coefficients in Uncertain Models

Stability-Adjusted Cross-Validation for Sparse Linear Regression

Provably Auditing Ordinary Least Squares in Low Dimensions

Robustness Auditing for Linear Regression: To Singularity and Beyond

Weighted least squares regression with the best robustness and high computability

Branch-and-bound algorithm for efficient reliability analysis of general coherent systems

A Bound on the Maximal Marginal Degrees of Freedom

A Statistical Perspective on Linear Programs with Uncertain Parameters

Data-driven confidence bound for structural response using segmented least squares: a mixed-integer programming approach

Cubature-based uncertainty estimation for nonlinear regression models

Robust confidence regions for the index and functional coefficients in the single-index varying coefficients regression model

A Perturbation Method for Inference on Regularized Regression Estimates

Adaptive estimation of the rank of the coefficient matrix in high dimensional multivariate response regression models

A Novel Framework for Improving the Breakdown Point of Robust Regression Algorithms

A robust and efficient change point detection method for high-dimensional linear models

Instability of Variable-selection Algorithms Used to Identify True Predictors of an Outcome in Intermediate-dimension Epidemiologic Studies

Augmented balancing weights as linear regression

Practical globally optimal consensus maximization by Branch-and-bound based on interval arithmetic

Testing a Single Regression Coefficient in High Dimensional Linear Models

Sparsity and Stability for Minimum-Variance Portfolios

When can Regression-Adjusted Control Variates Help? Rare Events, Sobolev Embedding and Minimax Optimality