Active Learning (AL) deals with identifying the most informative samples for labeling to reduce data annotation costs for supervised learning tasks. AL research suffers from the fact that lifts from literature generalize poorly and that only a small number of repetitions of experiments are conducted. To overcome these obstacles, we propose \emph{CDALBench}, the first active learning benchmark which includes tasks in computer vision, natural language processing and tabular learning. Furthermore, by providing an efficient, greedy oracle, \emph{CDALBench} can be evaluated with 50 runs for each experiment. We show, that both the cross-domain character and a large amount of repetitions are crucial for sophisticated evaluation of AL research. Concretely, we show that the superiority of specific methods varies over the different domains, making it important to evaluate Active Learning with a cross-domain benchmark. Additionally, we show that having a large amount of runs is crucial. With only conducting three runs as often done in the literature, the superiority of specific methods can strongly vary with the specific runs. This effect is so strong, that, depending on the seed, even a well-established method's performance can be significantly better and significantly worse than random for the same dataset.

该研究解决了主动学习（AL）在文献中通用性差和实验重复性不足的问题。提出的CDALBench是首个包含计算机视觉、自然语言处理和表格学习任务的主动学习基准，且提供了有效的贪婪oracle，允许每个实验执行50次，显著提升了评估AL研究的可靠性。研究表明，不同领域的主动学习方法表现差异显著，强调了使用跨领域基准的重要性。

跨领域主动学习基准