Abstract:Given a high-dimensional covariate matrix and a response vector, ridge-regularized sparse linear regression selects a subset of features that explains the relationship between covariates and the response in an interpretable manner. To select the sparsity and robustness of linear regressors, techniques like k-fold cross-validation are commonly used for hyperparameter tuning. However, cross-validation substantially increases the computational cost of sparse regression as it requires solving many mixed-integer optimization problems (MIOs). Additionally, validation metrics often serve as noisy estimators of test set errors, with different hyperparameter combinations leading to models with different noise levels. Therefore, optimizing over these metrics is vulnerable to out-of-sample disappointment, especially in underdetermined settings. To improve upon this state of affairs, we make two key contributions. First, motivated by the generalization theory literature, we propose selecting hyperparameters that minimize a weighted sum of a cross-validation metric and a model's output stability, thus reducing the risk of poor out-of-sample performance. Second, we leverage ideas from the mixed-integer optimization literature to obtain computationally tractable relaxations of k-fold cross-validation metrics and the output stability of regressors, facilitating hyperparameter selection after solving fewer MIOs. These relaxations result in an efficient cyclic coordinate descent scheme, achieving lower validation errors than via traditional methods such as grid search. On synthetic datasets, our confidence adjustment procedure improves out-of-sample performance by 2%-5% compared to minimizing the k-fold error alone. On 13 real-world datasets, our confidence adjustment procedure reduces test set error by 2%, on average.

CROSS-VALIDATED LOCAL LINEAR NONPARAMETRIC REGRESSION

Local Linear M‐estimation in Non‐parametric Spatial Regression

Cross-validatory Framework for Optimal Parameter Estimation of KPCA and KPLS Models

CROSS-VALIDATION AND MEDIAN CRITERION

Confidence Intervals Based on Local Linear Smoother

Local Linear Smoothers Using Asymmetric Kernels

Multivariate Universal Local Linear Kernel Estimators in Nonparametric Regression: Uniform Consistency

The Local Linear <path id="x1D440" d="M998 650l-8 -28q-71 -4 -86 -16t-22 -69l-50 -397q-3 -28 -4.5 -44t2 -29t6.5 -18.5t17 -10.5t24.5 -6.5t37.5 -3.5l-8 -28h-271l7 28q63 6 78 22t25 90l60 415h-2l-353 -552h-23l-130 536h-2l-70 -254q-44 -158 -47 -188q-5 -38 9 -51t71 -18l-6 -28h-241l8 28

Local linear smoothing for regression surfaces on the simplex using Dirichlet kernels

Robust Regularized Kernel Regression.

Local Linear Forests

On Neighbourhood Cross Validation

The Performance of Cross-Validation and Maximum Likelihood Estimators of Spline Smoothing Parameters

A new RKHS-based global testing for functional linear model

Local Linear Multiple Regression with Variable Bandwidth in the Presence of Heteroscedasticity

A generalized approximate cross validation for smoothing splines with non-Gaussian data

Cross-validation in nonparametric regression with outliers

Empirical Likelihood Confidence Intervals for Local Linear Smoothers

On the Asymptotic Optimality of Cross-Validation based Hyper-parameter Estimators for Regularized Least Squares Regression Problems

Stability-Adjusted Cross-Validation for Sparse Linear Regression

Optimal and Superoptimal Convergence Rate of the Local Linear Estimator of Nonparametric Regression Function in Continuous Time