Abstract:Given a high-dimensional covariate matrix and a response vector, ridge-regularized sparse linear regression selects a subset of features that explains the relationship between covariates and the response in an interpretable manner. To select the sparsity and robustness of linear regressors, techniques like k-fold cross-validation are commonly used for hyperparameter tuning. However, cross-validation substantially increases the computational cost of sparse regression as it requires solving many mixed-integer optimization problems (MIOs). Additionally, validation metrics often serve as noisy estimators of test set errors, with different hyperparameter combinations leading to models with different noise levels. Therefore, optimizing over these metrics is vulnerable to out-of-sample disappointment, especially in underdetermined settings. To improve upon this state of affairs, we make two key contributions. First, motivated by the generalization theory literature, we propose selecting hyperparameters that minimize a weighted sum of a cross-validation metric and a model's output stability, thus reducing the risk of poor out-of-sample performance. Second, we leverage ideas from the mixed-integer optimization literature to obtain computationally tractable relaxations of k-fold cross-validation metrics and the output stability of regressors, facilitating hyperparameter selection after solving fewer MIOs. These relaxations result in an efficient cyclic coordinate descent scheme, achieving lower validation errors than via traditional methods such as grid search. On synthetic datasets, our confidence adjustment procedure improves out-of-sample performance by 2%-5% compared to minimizing the k-fold error alone. On 13 real-world datasets, our confidence adjustment procedure reduces test set error by 2%, on average.

Validation design I: construction of validation designs via kernel herding

Validation of Machine Learning Prediction Models

Cross-validatory Framework for Optimal Parameter Estimation of KPCA and KPLS Models

An Adaptive Sequential Experiment Design Method for Model Validation

Optimal Design of Validation Experiments for the Prediction of Quantities of Interest

Bagging by Design (on the Suboptimality of Bagging)

Design of a Multiple Kernel Learning Algorithm for LS-SVM by Convex Programming

Consistent Validation for Predictive Methods in Spatial Settings

Designing Ensemble Learning Algorithms Using Kernel Methods

Stability-Adjusted Cross-Validation for Sparse Linear Regression

Proximal Validation Protocol

Optimal Design of Validation Experiment for Material Deterioration

Unifying Design-based Inference: On Bounding and Estimating the Variance of any Linear Estimator in any Experimental Design

Beware of Validation by Eye: Visual Validation of Linear Trends in Scatterplots

Extending Kernel Testing To General Designs

Jittering and Clustering: Strategies for the Construction of Robust Designs

CVXSADes: a stochastic algorithm for constructing optimal exact regression designs with single or multiple objectives

On Neighbourhood Cross Validation

Clustering-Based Validation Splits for Model Selection under Domain Shift

On Kernel Design for Regularized Non-Causal System Identification