Active Learning (AL) has gained prominence in integrating data-intensive
machine learning (ML) models into domains with limited labeled data. However,
its effectiveness diminishes significantly when the labeling budget is low. In
this paper, we first empirically observe the performance degradation of
existing AL algorithms in the low-budget settings, and then introduce Direct
Acquisition Optimization (DAO), a novel AL algorithm that optimizes sample
selections based on expected true loss reduction. Specifically, DAO utilizes
influence functions to update model parameters and incorporates an additional
acquisition strategy to mitigate bias in loss estimation. This approach
facilitates a more accurate estimation of the overall error reduction, without
extensive computations or reliance on labeled data. Experiments demonstrate
DAO's effectiveness in low budget settings, outperforming state-of-the-arts
approaches across seven benchmarks.

在本文中，我们首先通过实证观察到现有的主动学习算法在低预算环境下的性能下降，然后介绍了一种名为直接获取优化（DAO）的新型主动学习算法，该算法基于期望真实损失减少来优化样本选择。具体而言，DAO 利用影响函数来更新模型参数，并结合了一种附加的获取策略来减小损失估计中的偏差。该方法能够更准确地估计整体错误减少，而无需进行大量计算或依赖标记数据。实验结果表明，在低预算环境下，DAO 在七项基准测试中超越了现有方法。

低预算主动学习的直接获取优化

Direct Acquisition Optimization for Low-Budget Active Learning

The progress introduced by pre-trained language models and their fine-tuning
has resulted in significant improvements in most downstream NLP tasks. The
unsupervised training of a language model combined with further target task
fine-tuning has become the standard QA fine-tuning procedure. In this work, we
demonstrate that this strategy is sub-optimal for fine-tuning QA models,
especially under a low QA annotation budget, which is a usual setting in
practice due to the extractive QA labeling cost. We draw our conclusions by
conducting an exhaustive analysis of the performance of the alternatives of the
sequential fine-tuning strategy on different QA datasets. Based on the
experiments performed, we observed that the best strategy to fine-tune the QA
model in low-budget settings is taking a pre-trained language model (PLM) and
then fine-tuning PLM with a dataset composed of the target dataset and SQuAD
dataset. With zero extra annotation effort, the best strategy outperforms the
standard strategy by 2.28% to 6.48%. Our experiments provide one of the first
investigations on how to best fine-tune a QA system under a low budget and are
therefore of the utmost practical interest to the QA practitioners.

在本研究中，我们通过对不同 QA 数据集上的顺序微调策略的性能进行全面分析，证明了传统的基于预训练语言模型和微调策略在低预算设置下是次优的。我们的实验结果表明，在低预算设置下，最佳策略是使用预训练语言模型，并将其与目标数据集和 SQuAD 数据集进行微调。在不额外标注的情况下，该策略的性能优于标准策略 2.28% 至 6.48%。这些实验结果对于 QA 从业者在低预算下如何最好地微调 QA 系统具有极大的实用价值。