Abstract:Cross-validation is a common method for estimating the predictive performance of machine learning models. In a data-scarce regime, where one typically wishes to maximize the number of instances used for training the model, an approach called "leave-one-out cross-validation" is often used. In this design, a separate model is built for predicting each data instance after training on all other instances. Since this results in a single test data point available per model trained, predictions are aggregated across the entire dataset to calculate common rank-based performance metrics such as the area under the receiver operating characteristic or precision-recall curves. In this work, we demonstrate that this approach creates a negative correlation between the average label of each training fold and the label of its corresponding test instance, a phenomenon that we term distributional bias. As machine learning models tend to regress to the mean of their training data, this distributional bias tends to negatively impact performance evaluation and hyperparameter optimization. We show that this effect generalizes to leave-P-out cross-validation and persists across a wide range of modeling and evaluation approaches, and that it can lead to a bias against stronger regularization. To address this, we propose a generalizable rebalanced cross-validation approach that corrects for distributional bias. We demonstrate that our approach improves cross-validation performance evaluation in synthetic simulations and in several published leave-one-out analyses.

A leave-one-out cross validation bound for kernel methods with applications in learning

Generalized Convexity-Based Inexact Projection Method for Multiple Kernel Learning

Learning Kernels with Upper Bounds of Leave-One-out Error

The Optimality of Kernel Classifiers in Sobolev Space

Generalization in Kernel Regression Under Realistic Assumptions

Online Kernel Learning with a Near Optimal Sparsity Bound

Leave-one-out Bounds for Support Vector Regression

Distributional bias compromises leave-one-out cross-validation

Preventing Over-Fitting of Cross-Validation with Kernel Stability.

Theoretical Analysis of Leave-one-out Cross Validation for Non-differentiable Penalties under High-dimensional Settings

Infinite Kernel Learning: Generalization Bounds and Algorithms.

Provably Efficient Kernelized Q-Learning

Approximate Leave-one-out Cross Validation for Regression with $\ell_1$ Regularizers (extended version)

Cross-validation in nonparametric regression with outliers

Cross-validation on extreme regions

A Leave-One-Out Bound for nu-Support Vector Regression

Efficient Approximation of Cross-Validation for Kernel Methods Using Bouligand Influence Function

Kernel Stability for Model Selection in Kernel-Based Algorithms.

Regret Bounds for Online Kernel Selection in Continuous Kernel Space

Fast Cross-Validation for Kernel-Based Algorithms.