Many state-of-the-art automated machine learning (AutoML) systems use greedy
ensemble selection (GES) by Caruana et al. (2004) to ensemble models found
during model selection post hoc. Thereby, boosting predictive performance and
likely following Auto-Sklearn 1's insight that alternatives, like stacking or
gradient-free numerical optimization, overfit. Overfitting in Auto-Sklearn 1 is
much more likely than in other AutoML systems because it uses only low-quality
validation data for post hoc ensembling. Therefore, we were motivated to
analyze whether Auto-Sklearn 1's insight holds true for systems with
higher-quality validation data. Consequently, we compared the performance of
covariance matrix adaptation evolution strategy (CMA-ES), state-of-the-art
gradient-free numerical optimization, to GES on the 71 classification datasets
from the AutoML benchmark for AutoGluon. We found that Auto-Sklearn's insight
depends on the chosen metric. For the metric ROC AUC, CMA-ES overfits
drastically and is outperformed by GES -- statistically significantly for
multi-class classification. For the metric balanced accuracy, CMA-ES does not
overfit and outperforms GES significantly. Motivated by the successful
application of CMA-ES for balanced accuracy, we explored methods to stop CMA-ES
from overfitting for ROC AUC. We propose a method to normalize the weights
produced by CMA-ES, inspired by GES, that avoids overfitting for CMA-ES and
makes CMA-ES perform better than or similar to GES for ROC AUC.

本文研究了自动化机器学习中的模型集成方法，比较了不同方法在不同的度量标准下的性能表现，探讨了如何避免在使用数值优化方法时产生过拟合。

使用 CMA-ES 进行后续集成的 AutoML：成就与挽救失败

CMA-ES for Post Hoc Ensembling in AutoML: A Great Success and  Salvageable Failure

Creating large-scale high-quality labeled datasets is a major bottleneck in
supervised machine learning workflows. Auto-labeling systems are a promising
way to reduce reliance on manual labeling for dataset construction.
Threshold-based auto-labeling, where validation data obtained from humans is
used to find a threshold for confidence above which the data is
machine-labeled, is emerging as a popular solution used widely in practice.
Given the long shelf-life and diverse usage of the resulting datasets,
understanding when the data obtained by such auto-labeling systems can be
relied on is crucial. In this work, we analyze threshold-based auto-labeling
systems and derive sample complexity bounds on the amount of human-labeled
validation data required for guaranteeing the quality of machine-labeled data.
Our results provide two insights. First, reasonable chunks of the unlabeled
data can be automatically and accurately labeled by seemingly bad models.
Second, a hidden downside of threshold-based auto-labeling systems is
potentially prohibitive validation data usage. Together, these insights
describe the promise and pitfalls of using such systems. We validate our
theoretical guarantees with simulations and study the efficacy of
threshold-based auto-labeling on real datasets.

本文分析了基于阈值的自动标注系统，推导了保证机器标注数据质量所需的人工标注验证数据量的样本复杂度上限，并通过模拟和实际数据集研究了基于阈值的自动标注在真实数据上的有效性和使用限制。

差模型的好数据：基于阈值的自动标注的基础

Good Data from Bad Models : Foundations of Threshold-based Auto-labeling

Research in natural language processing proceeds, in part, by demonstrating
that new models achieve superior performance (e.g., accuracy) on held-out test
data, compared to previous results. In this paper, we demonstrate that test-set
performance scores alone are insufficient for drawing accurate conclusions
about which model performs best. We argue for reporting additional details,
especially performance on validation data obtained during model development. We
present a novel technique for doing so: expected validation performance of the
best-found model as a function of computation budget (i.e., the number of
hyperparameter search trials or the overall training time). Using our approach,
we find multiple recent model comparisons where authors would have reached a
different conclusion if they had used more (or less) computation. Our approach
also allows us to estimate the amount of computation required to obtain a given
accuracy; applying it to several recently published results yields massive
variation across papers, from hours to weeks. We conclude with a set of best
practices for reporting experimental results which allow for robust future
comparisons, and provide code to allow researchers to use our technique.

通过使用验证数据和计算预算等指标，本文提出了一种新的模型比较方法，从而帮助实现更好地汇报实验结果和确保未来准确对比。