Label aggregation such as majority voting is commonly used to resolve annotator disagreement in dataset creation. However, this may disregard minority values and opinions. Recent studies indicate that learning from individual annotations outperforms learning from aggregated labels, though they require a considerable amount of annotation. Active learning, as an annotation cost-saving strategy, has not been fully explored in the context of learning from disagreement. We show that in the active learning setting, a multi-head model performs significantly better than a single-head model in terms of uncertainty estimation. By designing and evaluating acquisition functions with annotator-specific heads on two datasets, we show that group-level entropy works generally well on both datasets. Importantly, it achieves performance in terms of both prediction and uncertainty estimation comparable to full-scale training from disagreement, while saving up to 70% of the annotation budget.

在主动学习设置中，多头模型在不确定性估计方面显着优于单头模型。通过设计和评估具有特定标注员头的获取函数在两个数据集上，我们展示了群体级别的熵在两个数据集上表现良好。重要的是，它在预测和不确定性估计方面的性能与全量训练相当，同时节约高达70%的标注预算。

ACTOR：利用注释者特定的分类器头部进行主动学习以应对人类标签变异