Active learning allows machine learning models to be trained using fewer
labels while retaining similar performance to traditional supervised learning.
An active learner selects the most informative data points, requests their
labels, and retrains itself. While this approach is promising, it raises the
question of how to determine when the model is `good enough' without the
additional labels required for traditional evaluation. Previously, different
stopping criteria have been proposed aiming to identify the optimal stopping
point. Yet, optimality can only be expressed as a domain-dependent trade-off
between accuracy and the number of labels, and no criterion is superior in all
applications. As a further complication, a comparison of criteria for a
particular real-world application would require practitioners to collect
additional labelled data they are aiming to avoid by using active learning in
the first place. This work enables practitioners to employ active learning by
providing actionable recommendations for which stopping criteria are best for a
given real-world scenario. We contribute the first large-scale comparison of
stopping criteria for pool-based active learning, using a cost measure to
quantify the accuracy/label trade-off, public implementations of all stopping
criteria we evaluate, and an open-source framework for evaluating stopping
criteria. Our research enables practitioners to substantially reduce labelling
costs by utilizing the stopping criterion which best suits their domain.

本文介绍利用主动学习通过提供可行建议，确定适合特定现实情景的停止标准以减少标注成本的方法。作者提出了大规模的停止标准比较，使用成本测量来量化准确性 / 标签交换，并提供了所有停止标准的公共实现和一个开源框架。