To accurately capture the variability in human judgments for subjective NLP
tasks, incorporating a wide range of perspectives in the annotation process is
crucial. Active Learning (AL) addresses the high costs of collecting human
annotations by strategically annotating the most informative samples. We
introduce Annotator-Centric Active Learning (ACAL), which incorporates an
annotator selection strategy following data sampling. Our objective is
two-fold: (1) to efficiently approximate the full diversity of human judgments,
and to assess model performance using annotator-centric metrics, which
emphasize minority perspectives over a majority. We experiment with multiple
annotator selection strategies across seven subjective NLP tasks, employing
both traditional and novel, human-centered evaluation metrics. Our findings
indicate that ACAL improves data efficiency and excels in annotator-centric
performance evaluations. However, its success depends on the availability of a
sufficiently large and diverse pool of annotators to sample from.

利用主观自然语言处理任务进行准确的人类判断的关键是在注释过程中纳入广泛的视角。引入了注释者中心主动学习策略 (ACAL)，结合数据采样和注释者选择策略，旨在高效地近似获取人类判断的全面多样性，并使用注释者中心度量评估模型性能。对于七个主观自然语言处理任务进行了多种注释者选择策略的实验，同时采用了传统和新颖的以人为中心的评估指标。结果表明，ACAL 提高了数据效率并在注释者中心度量评估中表现出色，但其成功仍取决于足够大而多样的注释者样本池的可用性。

面向主观 NLP 任务的标注者中心主动学习

Annotator-Centric Active Learning for Subjective NLP Tasks

In subjective NLP tasks, where a single ground truth does not exist, the
inclusion of diverse annotators becomes crucial as their unique perspectives
significantly influence the annotations. In realistic scenarios, the annotation
budget often becomes the main determinant of the number of perspectives (i.e.,
annotators) included in the data and subsequent modeling. We introduce a novel
framework for annotation collection and modeling in subjective tasks that aims
to minimize the annotation budget while maximizing the predictive performance
for each annotator. Our framework has a two-stage design: first, we rely on a
small set of annotators to build a multitask model, and second, we augment the
model for a new perspective by strategically annotating a few samples per
annotator. To test our framework at scale, we introduce and release a unique
dataset, Moral Foundations Subjective Corpus, of 2000 Reddit posts annotated by
24 annotators for moral sentiment. We demonstrate that our framework surpasses
the previous SOTA in capturing the annotators' individual perspectives with as
little as 25% of the original annotation budget on two datasets. Furthermore,
our framework results in more equitable models, reducing the performance
disparity among annotators.

通过使用多样的标注员并最大限度降低标注预算，我们提出了一个新的主观任务注释收集和建模框架，通过两阶段设计，使用少量标注员和有策略地标注一些样本，提高了预测性能，并减少了注释预算，同时我们构建了一个包含 2000 个 Reddit 帖子的独特数据集，并在两个数据集上展示了我们的框架超越之前的最优结果，捕捉了标注员个体视角并减少了性能差异。

通过少样本注释者适应实现成本高效的主观任务注释和建模

Cost-Efficient Subjective Task Annotation and Modeling through Few-Shot  Annotator Adaptation

Annotators' sociodemographic backgrounds (i.e., the individual compositions
of their gender, age, educational background, etc.) have a strong impact on
their decisions when working on subjective NLP tasks, such as hate speech
detection. Often, heterogeneous backgrounds result in high disagreements. To
model this variation, recent work has explored sociodemographic prompting, a
technique, which steers the output of prompt-based models towards answers that
humans with specific sociodemographic profiles would give. However, the
available NLP literature disagrees on the efficacy of this technique -- it
remains unclear, for which tasks and scenarios it can help and evaluations are
limited to specific tasks only. We address this research gap by presenting the
largest and most comprehensive study of sociodemographic prompting today.
Concretely, we evaluate several prompt formulations across seven datasets and
six instruction-tuned model families. We find that (1) while sociodemographic
prompting can be beneficial for improving zero-shot learning in subjective NLP
tasks, (2) its outcomes largely vary for different model types, sizes, and
datasets, (3) are subject to large variance with regards to prompt
formulations. Thus, sociodemographic prompting is not a reliable proxy for
traditional data annotation with a sociodemographically heterogeneous group of
annotators. Instead, we propose (4) to use it for identifying ambiguous
instances resulting in more informed annotation efforts.

注释者的社会人口背景对主观 NLP 任务的决策产生很大影响。社会人口提示是一种技术，用于将基于提示的模型的输出指引到具有特定社会人口特征的人会给出的答案。本研究通过对七个数据集和六个模型系列进行评估，得出了几个关键发现，并提出使用社会人口提示来识别不明确的实例，从而进行更加明智的注释工作。