Classification algorithms aim to predict an unknown label (e.g., a quality
class) for a new instance (e.g., a product). Therefore, training samples
(instances and labels) are used to deduct classification hypotheses. Often, it
is relatively easy to capture instances but the acquisition of the
corresponding labels remain difficult or expensive. Active learning algorithms
select the most beneficial instances to be labeled to reduce cost. In research,
this labeling procedure is simulated and therefore a ground truth is available.
But during deployment, active learning is a one-shot problem and an evaluation
set is not available. Hence, it is not possible to reliably estimate the
performance of the classification system during learning and it is difficult to
decide when the system fulfills the quality requirements (stopping criteria).
In this article, we formalize the task and review existing strategies to assess
the performance of an actively trained classifier during training. Furthermore,
we identified three major challenges: 1)~to derive a performance distribution,
2)~to preserve representativeness of the labeled subset, and 3) to correct
against sampling bias induced by an intelligent selection strategy. In a
qualitative analysis, we evaluate different existing approaches and show that
none of them reliably estimates active learning performance stating a major
challenge for future research for such systems. All plots and experiments are
provided in a Jupyter notebook that is available for download.

本文研究了如何在不可靠的情况下评估主动学习算法在训练期间的性能，并提出了三个主要挑战：如何推导性能分布，如何保留标记子集的代表性，以及如何纠正由于智能选择策略而引起的抽样偏差。

评估实时主动学习绩效的限制

Limitations of Assessing Active Learning Performance at Runtime

We present a practical and statistically consistent scheme for actively
learning binary classifiers under general loss functions. Our algorithm uses
importance weighting to correct sampling bias, and by controlling the variance,
we are able to give rigorous label complexity bounds for the learning process.
Experiments on passively labeled data show that this approach reduces the label
complexity required to achieve good predictive performance on many learning
problems.

提供了一个实用的、统计上连贯的方案，可在通用损失函数下主动学习二分类器，该算法使用重要性加权来纠正抽样偏差，并通过控制变量来给出严格的标签复杂性界限，实验表明，该方法减少了实现许多学习问题的良好预测性能所需的标签复杂性。