In subjective NLP tasks, where a single ground truth does not exist, the
inclusion of diverse annotators becomes crucial as their unique perspectives
significantly influence the annotations. In realistic scenarios, the annotation
budget often becomes the main determinant of the number of perspectives (i.e.,
annotators) included in the data and subsequent modeling. We introduce a novel
framework for annotation collection and modeling in subjective tasks that aims
to minimize the annotation budget while maximizing the predictive performance
for each annotator. Our framework has a two-stage design: first, we rely on a
small set of annotators to build a multitask model, and second, we augment the
model for a new perspective by strategically annotating a few samples per
annotator. To test our framework at scale, we introduce and release a unique
dataset, Moral Foundations Subjective Corpus, of 2000 Reddit posts annotated by
24 annotators for moral sentiment. We demonstrate that our framework surpasses
the previous SOTA in capturing the annotators' individual perspectives with as
little as 25% of the original annotation budget on two datasets. Furthermore,
our framework results in more equitable models, reducing the performance
disparity among annotators.

通过使用多样的标注员并最大限度降低标注预算，我们提出了一个新的主观任务注释收集和建模框架，通过两阶段设计，使用少量标注员和有策略地标注一些样本，提高了预测性能，并减少了注释预算，同时我们构建了一个包含 2000 个 Reddit 帖子的独特数据集，并在两个数据集上展示了我们的框架超越之前的最优结果，捕捉了标注员个体视角并减少了性能差异。