State-of-the-art automated machine learning systems for tabular data often
employ cross-validation; ensuring that measured performances generalize to
unseen data, or that subsequent ensembling does not overfit. However, using
k-fold cross-validation instead of holdout validation drastically increases the
computational cost of validating a single configuration. While ensuring better
generalization and, by extension, better performance, the additional cost is
often prohibitive for effective model selection within a time budget. We aim to
make model selection with cross-validation more effective. Therefore, we study
early stopping the process of cross-validation during model selection. We
investigate the impact of early stopping on random search for two algorithms,
MLP and random forest, across 36 classification datasets. We further analyze
the impact of the number of folds by considering 3-, 5-, and 10-folds. In
addition, we investigate the impact of early stopping with Bayesian
optimization instead of random search and also repeated cross-validation. Our
exploratory study shows that even a simple-to-understand and easy-to-implement
method consistently allows model selection to converge faster; in ~94% of all
datasets, on average by ~214%. Moreover, stopping cross-validation enables
model selection to explore the search space more exhaustively by considering
+167% configurations on average within one hour, while also obtaining better
overall performance.

使用早停法进行交叉验证的模型选择，可以更高效地收敛，涵盖更多搜索空间，并实现更好的性能。