Abstract:Given a high-dimensional covariate matrix and a response vector, ridge-regularized sparse linear regression selects a subset of features that explains the relationship between covariates and the response in an interpretable manner. To select the sparsity and robustness of linear regressors, techniques like k-fold cross-validation are commonly used for hyperparameter tuning. However, cross-validation substantially increases the computational cost of sparse regression as it requires solving many mixed-integer optimization problems (MIOs). Additionally, validation metrics often serve as noisy estimators of test set errors, with different hyperparameter combinations leading to models with different noise levels. Therefore, optimizing over these metrics is vulnerable to out-of-sample disappointment, especially in underdetermined settings. To improve upon this state of affairs, we make two key contributions. First, motivated by the generalization theory literature, we propose selecting hyperparameters that minimize a weighted sum of a cross-validation metric and a model's output stability, thus reducing the risk of poor out-of-sample performance. Second, we leverage ideas from the mixed-integer optimization literature to obtain computationally tractable relaxations of k-fold cross-validation metrics and the output stability of regressors, facilitating hyperparameter selection after solving fewer MIOs. These relaxations result in an efficient cyclic coordinate descent scheme, achieving lower validation errors than via traditional methods such as grid search. On synthetic datasets, our confidence adjustment procedure improves out-of-sample performance by 2%-5% compared to minimizing the k-fold error alone. On 13 real-world datasets, our confidence adjustment procedure reduces test set error by 2%, on average.

Explicit solutions for the asymptotically-optimal bandwidth in cross-validation

A weighted least-squares cross-validation bandwidth selector for kernel density estimation

Asymptotic Behavior of Bandwidth Selected by the Cross-Validation Method for Local Polynomial Fitting

Cross-validatory Framework for Optimal Parameter Estimation of KPCA and KPLS Models

Bagging cross-validated bandwidths with application to Big Data

Cross-validation in nonparametric regression with outliers

Novel kernel density estimator based on ensemble unbiased cross-validation

On the Asymptotic Optimality of Cross-Validation based Hyper-parameter Estimators for Regularized Least Squares Regression Problems

Multivariate bandwidth selection for local linear regression

Sequential Cross-Validated Bandwidth Selection Under Dependence and Anscombe-Type Extensions to Random Time Horizons

Variable bandwidth kernel regression estimation

Stability-Adjusted Cross-Validation for Sparse Linear Regression

Multivariate Plug-in Bandwidth for Local Linear Regression

Cross-validation in high-dimensional spaces: a lifeline for least-squares models and multi-class LDA

Cross-validation on extreme regions

Local Linear Multiple Regression with Variable Bandwidth in the Presence of Heteroscedasticity

Asymptotic Theory of Least Distances Estimate in Multivariate Linear Models

On Neighbourhood Cross Validation

Unified treatment of the asymptotics of asymmetric kernel density estimators

One-Sided Cross-Validation for Nonsmooth Density Functions