Abstract:State-of-the-art automated machine learning systems for tabular data often employ cross-validation; ensuring that measured performances generalize to unseen data, or that subsequent ensembling does not overfit. However, using k-fold cross-validation instead of holdout validation drastically increases the computational cost of validating a single configuration. While ensuring better generalization and, by extension, better performance, the additional cost is often prohibitive for effective model selection within a time budget. We aim to make model selection with cross-validation more effective. Therefore, we study early stopping the process of cross-validation during model selection. We investigate the impact of early stopping on random search for two algorithms, MLP and random forest, across 36 classification datasets. We further analyze the impact of the number of folds by considering 3-, 5-, and 10-folds. In addition, we investigate the impact of early stopping with Bayesian optimization instead of random search and also repeated cross-validation. Our exploratory study shows that even a simple-to-understand and easy-to-implement method consistently allows model selection to converge faster; in ~94% of all datasets, on average by ~214%. Moreover, stopping cross-validation enables model selection to explore the search space more exhaustively by considering +167% configurations on average within one hour, while also obtaining better overall performance.
What problem does this paper attempt to address?
The problem that this paper attempts to solve is the high - cost and time - consuming problem in model selection using cross - validation in AutoML (Automated Machine Learning) systems. Specifically, compared with holdout validation, using \(k\)-fold cross - validation can improve the generalization ability of model performance, but at the same time, it will significantly increase the computational cost of validating a single configuration. This additional cost often limits effective model selection within the time budget, because better configurations may not be evaluated within the budgeted time, or the convergence speed of model selection on the validation data becomes slower.
To solve this problem, the paper proposes a method to stop cross - validation early during the model selection process, aiming to make model selection under cross - validation more efficient. By studying the impact of early stopping on random search, analyzing the impact of different numbers of folds (such as 3 - fold, 5 - fold, and 10 - fold), and the effect of combining early stopping with Bayesian Optimization (BO) and repeated cross - validation, the paper explores how the early - stopping strategy can accelerate the convergence of model selection, while allowing a broader exploration of the search space, and ultimately achieving better overall performance.
The main contribution of the paper lies in providing evidence of early - stopping methods that are simple to understand, easy to implement, and perform well. These methods can accelerate on average by 214% on about 94% of the datasets, and on average consider more configurations (increased by 167%) within the same time budget, while obtaining better overall performance. In addition, the paper also provides a reproducible and extensible research framework for future research on early - stopping cross - validation.