To accurately capture the variability in human judgments for subjective NLP
tasks, incorporating a wide range of perspectives in the annotation process is
crucial. Active Learning (AL) addresses the high costs of collecting human
annotations by strategically annotating the most informative samples. We
introduce Annotator-Centric Active Learning (ACAL), which incorporates an
annotator selection strategy following data sampling. Our objective is
two-fold: (1) to efficiently approximate the full diversity of human judgments,
and to assess model performance using annotator-centric metrics, which
emphasize minority perspectives over a majority. We experiment with multiple
annotator selection strategies across seven subjective NLP tasks, employing
both traditional and novel, human-centered evaluation metrics. Our findings
indicate that ACAL improves data efficiency and excels in annotator-centric
performance evaluations. However, its success depends on the availability of a
sufficiently large and diverse pool of annotators to sample from.

利用主观自然语言处理任务进行准确的人类判断的关键是在注释过程中纳入广泛的视角。引入了注释者中心主动学习策略 (ACAL)，结合数据采样和注释者选择策略，旨在高效地近似获取人类判断的全面多样性，并使用注释者中心度量评估模型性能。对于七个主观自然语言处理任务进行了多种注释者选择策略的实验，同时采用了传统和新颖的以人为中心的评估指标。结果表明，ACAL 提高了数据效率并在注释者中心度量评估中表现出色，但其成功仍取决于足够大而多样的注释者样本池的可用性。