Abstract:Given a high-dimensional covariate matrix and a response vector, ridge-regularized sparse linear regression selects a subset of features that explains the relationship between covariates and the response in an interpretable manner. To select the sparsity and robustness of linear regressors, techniques like k-fold cross-validation are commonly used for hyperparameter tuning. However, cross-validation substantially increases the computational cost of sparse regression as it requires solving many mixed-integer optimization problems (MIOs). Additionally, validation metrics often serve as noisy estimators of test set errors, with different hyperparameter combinations leading to models with different noise levels. Therefore, optimizing over these metrics is vulnerable to out-of-sample disappointment, especially in underdetermined settings. To improve upon this state of affairs, we make two key contributions. First, motivated by the generalization theory literature, we propose selecting hyperparameters that minimize a weighted sum of a cross-validation metric and a model's output stability, thus reducing the risk of poor out-of-sample performance. Second, we leverage ideas from the mixed-integer optimization literature to obtain computationally tractable relaxations of k-fold cross-validation metrics and the output stability of regressors, facilitating hyperparameter selection after solving fewer MIOs. These relaxations result in an efficient cyclic coordinate descent scheme, achieving lower validation errors than via traditional methods such as grid search. On synthetic datasets, our confidence adjustment procedure improves out-of-sample performance by 2%-5% compared to minimizing the k-fold error alone. On 13 real-world datasets, our confidence adjustment procedure reduces test set error by 2%, on average.

Algebraic Shortcuts for Leave-One-Out Cross-Validation in Supervised Network Inference

Cross-validation in high-dimensional spaces: a lifeline for least-squares models and multi-class LDA

Cross-Validation for Training and Testing Co-occurrence Network Inference Algorithms

Network cross-validation by edge sampling

Efficient strategies for leave-one-out cross validation for genomic best linear unbiased prediction

Fast Cross-Validation Algorithms for Least Squares Support Vector Machine and Kernel Ridge Regression

Is Cross-Validation the Gold Standard to Evaluate Model Performance?

Stability-Adjusted Cross-Validation for Sparse Linear Regression

Don't Waste Your Time: Early Stopping Cross-Validation

A swiss army infinitesimal jackknife

Bayesian nonparametric cross-study validation of prediction methods

Testing Conditional Independence in Supervised Learning Algorithms

Patch Shortcuts: Interpretable Proxy Models Efficiently Find Black-Box Vulnerabilities

Bootstrapping the Out-of-sample Predictions for Efficient and Accurate Cross-Validation

Automatic cross-validation in structured models: Is it time to leave out leave-one-out?

Bayesian leave-one-out cross-validation for large data

Efficient, adaptive cross-validation for tuning and comparing models, with application to drug discovery

Robust inference via knockoffs

Efficient leave-one-out cross-validation for Bayesian non-factorized normal and Student-t models

Distributional bias compromises leave-one-out cross-validation

Shortcut learning in deep neural networks