In settings where only a budgeted amount of labeled data can be afforded,
active learning seeks to devise query strategies for selecting the most
informative data points to be labeled, aiming to enhance learning algorithms'
efficiency and performance. Numerous such query strategies have been proposed
and compared in the active learning literature. However, the community still
lacks standardized benchmarks for comparing the performance of different query
strategies. This particularly holds for the combination of query strategies
with different learning algorithms into active learning pipelines and examining
the impact of the learning algorithm choice. To close this gap, we propose
ALPBench, which facilitates the specification, execution, and performance
monitoring of active learning pipelines. It has built-in measures to ensure
evaluations are done reproducibly, saving exact dataset splits and
hyperparameter settings of used algorithms. In total, ALPBench consists of 86
real-world tabular classification datasets and 5 active learning settings,
yielding 430 active learning problems. To demonstrate its usefulness and broad
compatibility with various learning algorithms and query strategies, we conduct
an exemplary study evaluating 9 query strategies paired with 8 learning
algorithms in 2 different settings. We provide ALPBench here:
this https URL

在仅有限标记数据可用的情况下，主动学习通过设计查询策略来选择信息量最大的数据点进行标记，旨在提高学习算法的效率和性能。然而，目前缺乏对不同查询策略性能进行比较的标准化基准，特别是在将不同学习算法与主动学习流程相结合并考察学习算法选择的影响方面。为了弥补这一空白，我们提出了 ALPBench，用于支持主动学习流程的规范、执行和性能监控。它内置了可确保可重复评估的措施，保存了使用算法的精确数据集拆分和超参数设置。总共，ALPBench 包含 86 个真实的表格分类数据集和 5 个主动学习设置，产生 430 个主动学习问题。为了证明其对各种学习算法和查询策略的有用性和广泛兼容性，我们进行了一项示例研究，评估了 9 种查询策略与 8 种学习算法在两种不同设置下的表现。我们在此提供 ALPBench：[URL]