Supervised machine learning and deep learning require a large amount of
labeled data, which data scientists obtain in a manual, and time-consuming
annotation process. To mitigate this challenge, Active Learning (AL) proposes
promising data points to annotators they annotate next instead of a subsequent
or random sample. This method is supposed to save annotation effort while
maintaining model performance. However, practitioners face many AL strategies
for different tasks and need an empirical basis to choose between them. Surveys
categorize AL strategies into taxonomies without performance indications.
Presentations of novel AL strategies compare the performance to a small subset
of strategies. Our contribution addresses the empirical basis by introducing a
reproducible active learning evaluation (ALE) framework for the comparative
evaluation of AL strategies in NLP. The framework allows the implementation of
AL strategies with low effort and a fair data-driven comparison through
defining and tracking experiment parameters (e.g., initial dataset size, number
of data points per query step, and the budget). ALE helps practitioners to make
more informed decisions, and researchers can focus on developing new, effective
AL strategies and deriving best practices for specific use cases. With best
practices, practitioners can lower their annotation costs. We present a case
study to illustrate how to use the framework.

使用可复现的主动学习评估框架来比较评估自然语言处理中的主动学习策略，并通过定义和跟踪实验参数，帮助实践者作出更明智的决策，同时帮助研究者开发出新的高效主动学习策略和制定最佳实践，从而降低注释成本。