Abstract:Given a high-dimensional covariate matrix and a response vector, ridge-regularized sparse linear regression selects a subset of features that explains the relationship between covariates and the response in an interpretable manner. To select the sparsity and robustness of linear regressors, techniques like k-fold cross-validation are commonly used for hyperparameter tuning. However, cross-validation substantially increases the computational cost of sparse regression as it requires solving many mixed-integer optimization problems (MIOs). Additionally, validation metrics often serve as noisy estimators of test set errors, with different hyperparameter combinations leading to models with different noise levels. Therefore, optimizing over these metrics is vulnerable to out-of-sample disappointment, especially in underdetermined settings. To improve upon this state of affairs, we make two key contributions. First, motivated by the generalization theory literature, we propose selecting hyperparameters that minimize a weighted sum of a cross-validation metric and a model's output stability, thus reducing the risk of poor out-of-sample performance. Second, we leverage ideas from the mixed-integer optimization literature to obtain computationally tractable relaxations of k-fold cross-validation metrics and the output stability of regressors, facilitating hyperparameter selection after solving fewer MIOs. These relaxations result in an efficient cyclic coordinate descent scheme, achieving lower validation errors than via traditional methods such as grid search. On synthetic datasets, our confidence adjustment procedure improves out-of-sample performance by 2%-5% compared to minimizing the k-fold error alone. On 13 real-world datasets, our confidence adjustment procedure reduces test set error by 2%, on average.

Preventing Over-Fitting of Cross-Validation with Kernel Stability.

Kernel Stability for Model Selection in Kernel-Based Algorithms.

Generalized Convexity-Based Inexact Projection Method for Multiple Kernel Learning

Cross-validatory Framework for Optimal Parameter Estimation of KPCA and KPLS Models

Kernel Selection with Spectral Perturbation Stability of Kernel Matrix

Stability-Adjusted Cross-Validation for Sparse Linear Regression

Fast Cross-Validation for Kernel-Based Algorithms.

Fast Cross-Validation Algorithms for Least Squares Support Vector Machine and Kernel Ridge Regression

Efficient Cross Validation for SVR Based on Center Distance in Kernel Space

Efficient Approximation of Cross-Validation for Kernel Methods Using Bouligand Influence Function

Stability and Generalization of Kernel Clustering: From Single Kernel to Multiple Kernel

Cross-validation in nonparametric regression with outliers

Kernel‐based Generalized Cross‐validation in Non‐parametric Mixed‐effect Models

Is Cross-Validation the Gold Standard to Evaluate Model Performance?

Robust Regularized Kernel Regression.

Eigenvalues Perturbation of Integral Operator for Kernel Selection.

Iterative Kernel Regression with Preconditioning

Novel kernel density estimator based on ensemble unbiased cross-validation

Failures and Successes of Cross-Validation for Early-Stopped Gradient Descent

Cross-validation on Extreme Regions

Efficient Cross-Validation for Semi-Supervised Learning