Abstract:Selecting a final machine learning (ML) model typically occurs after a process of hyperparameter optimization in which many candidate models with varying structural properties and algorithmic settings are evaluated and compared. Evaluating each candidate model commonly relies on k-fold cross validation, wherein the data are randomly subdivided into k folds, with each fold being iteratively used as a validation set for a model that has been trained using the remaining folds. While many research studies have sought to accelerate ML model selection by applying metaheuristic and other search methods to the hyperparameter space, no consideration has been given to the k-fold cross validation process itself as a means of rapidly identifying the best-performing model. The current study rectifies this oversight by introducing a greedy k-fold cross validation method and demonstrating that greedy k-fold cross validation can vastly reduce the average time required to identify the best-performing model when given a fixed computational budget and a set of candidate models. This improved search time is shown to hold across a variety of ML algorithms and real-world datasets. For scenarios without a computational budget, this paper also introduces an early stopping algorithm based on the greedy cross validation method. The greedy early stopping method is shown to outperform a competing, state-of-the-art early stopping method both in terms of search time and the quality of the ML models selected by the algorithm. Since hyperparameter optimization is among the most time-consuming, computationally intensive, and monetarily expensive tasks in the broader process of developing ML-based solutions, the ability to rapidly identify optimal machine learning models using greedy cross validation has obvious and substantial benefits to organizations and researchers alike.

Fast model selection by limiting SVM training times

Novel Linear Search for Support Vector Machine Parameter Selection

SVM KERNEL PARAMETER SELECTION BASED ON IMPROVED GSO

Feature Selection and Parameter Optimization for Support Vector Machines: A New Approach Based on Genetic Algorithm with Feature Chromosomes.

A Fast Training Method for OC-SVM Based on the Random Sampling Lemma

Fast and Efficient Strategies for Model Selection of Gaussian Support Vector Machine

On Selecting Effective Patterns for Fast Support Vector Regression Training.

Efficient Selection of Gaussian Kernel SVM Parameters for Imbalanced Data

A hybrid method for speeding SVM training

Simple, fast and accurate hyper-parameter tuning in Gaussian-kernel SVM

A Fast SVM-based Feature Selection Method

Don't Waste Your Time: Early Stopping Cross-Validation

Kernel Parameter Selection for Support Vector Machine Classification

Efficient Model Selection for Support Vector Machine with Gaussian Kernel Function

MASCOT: Fast and Highly Scalable SVM Cross-Validation Using GPUs and SSDs

SVM Parameter Optimization using Grid Search and Genetic Algorithm to Improve Classification Performance

Super-parameter Selection for Gaussian-Kernel SVM Based on Outlier-Resisting

A simple and reliable instance selection for fast training support vector machine: Valid Border Recognition

Gene Selection Using Gaussian Kernel Support Vector Machine Based Recursive Feature Elimination with Adaptive Kernel Width Strategy

MaxMin-L2-SVC-NCH: A Novel Approach for Support Vector Classifier Training and Parameter Selection

Greed Is Good: Rapid Hyperparameter Optimization and Model Selection Using Greedy k-Fold Cross Validation