Abstract:As a technique that can compactly represent complex patterns, machine learning has significant potential for predictive inference. K-fold cross-validation (CV) is the most common approach to ascertaining the likelihood that a machine learning outcome is generated by chance and frequently outperforms conventional hypothesis testing. This improvement uses measures directly obtained from machine learning classifications, such as accuracy, that do not have a parametric description. To approach a frequentist analysis within machine learning pipelines, a permutation test or simple statistics from data partitions (i.e. folds) can be added to estimate confidence intervals. Unfortunately, neither parametric nor non-parametric tests solve the inherent problems around partitioning small sample-size datasets and learning from heterogeneous data sources. The fact that machine learning strongly depends on the learning parameters and the distribution of data across folds recapitulates familiar difficulties around excess false positives and replication. The origins of this problem are demonstrated by simulating common experimental circumstances, including small sample sizes, low numbers of predictors, and heterogeneous data sources. A novel statistical test based on K-fold CV and the Upper Bound of the actual error (K-fold CUBV) is composed, where uncertain predictions of machine learning with CV are bounded by the \emph{worst case} through the evaluation of concentration inequalities. Probably Approximately Correct-Bayesian upper bounds for linear classifiers in combination with K-fold CV is used to estimate the empirical error. The performance with neuroimaging datasets suggests this is a robust criterion for detecting effects, validating accuracy values obtained from machine learning whilst avoiding excess false positives.

Sensitivity Analysis with Cross-Validation for Feature Selection and Manifold Learning

Cross-validatory Framework for Optimal Parameter Estimation of KPCA and KPLS Models

Efficient Cross-Validation for Semi-Supervised Learning

Cross-Validation Approach to Evaluate Clustering Algorithms: An Experimental Study Using Multi-Label Datasets

Is K-fold cross validation the best model selection method for Machine Learning?

Don't Waste Your Time: Early Stopping Cross-Validation

A survey of cross-validation procedures for model selection

Cross-validation: what does it estimate and how well does it do it?

Feature subset selection for support vector machines through sensitivity analysis

Estimating the prediction performance of spatial models via spatial k-fold cross validation

An Improved Cross-Validated Adversarial Validation Method.

On The Smoothness of Cross-Validation-Based Estimators Of Classifier Performance

Cross-validation of component models: A critical look at current methods

Cross-validation in high-dimensional spaces: a lifeline for least-squares models and multi-class LDA

Cross-Validation Is All You Need: A Statistical Approach To Label Noise Estimation

Monte Carlo Cross‐validation for Selecting a Model and Estimating the Prediction Error in Multivariate Calibration

Performance-Estimation Properties of Cross-Validation-Based Protocols with Simultaneous Hyper-Parameter Optimization

Empirical investigation of multi-source cross-validation in clinical machine learning

Model Selection Via Multifold Cross Validation

Impact Analysis of Classification Performance for Cross-Validation of Imbalance Spliting Data

Research of Machine Learning algorithms using K-fold cross validation