Longstanding data labeling practices in machine learning involve collecting
and aggregating labels from multiple annotators. But what should we do when
annotators disagree? Though annotator disagreement has long been seen as a
problem to minimize, new perspectivist approaches challenge this assumption by
treating disagreement as a valuable source of information. In this position
paper, we examine practices and assumptions surrounding the causes of
disagreement--some challenged by perspectivist approaches, and some that remain
to be addressed--as well as practical and normative challenges for work
operating under these assumptions. We conclude with recommendations for the
data labeling pipeline and avenues for future research engaging with
subjectivity and disagreement.

对于机器学习中的数据标注，多年来一直采用从多个标注者那里收集和汇总标签的方法。然而，当标注者不一致时，我们应该怎么办呢？尽管长期以来，标注者的不一致被视为需要最小化的问题，但新的观点主义方法质疑这一假设，将不一致视为宝贵的信息源。本文中，我们研究了有关标注者不一致的实践和假设，一些是受观点主义方法的挑战，一些是待解决的问题，以及基于这些假设进行工作的实际和规范性挑战。我们最后提出了对数据标注流程的建议，并探索与主观性和不一致性有关的未来研究方向。

透视者的范式转变：对捕捉人工标签的假设和挑战

The Perspectivist Paradigm Shift: Assumptions and Challenges of  Capturing Human Labels

There are two competing approaches for modelling annotator disagreement:
distributional soft-labelling approaches (which aim to capture the level of
disagreement) or modelling perspectives of individual annotators or groups
thereof. We adapt a multi-task architecture -- which has previously shown
success in modelling perspectives -- to evaluate its performance on the SEMEVAL
Task 11. We do so by combining both approaches, i.e. predicting individual
annotator perspectives as an interim step towards predicting annotator
disagreement. Despite its previous success, we found that a multi-task approach
performed poorly on datasets which contained distinct annotator opinions,
suggesting that this approach may not always be suitable when modelling
perspectives. Furthermore, our results explain that while strongly
perspectivist approaches might not achieve state-of-the-art performance
according to evaluation metrics used by distributional approaches, our approach
allows for a more nuanced understanding of individual perspectives present in
the data. We argue that perspectivist approaches are preferable because they
enable decision makers to amplify minority views, and that it is important to
re-evaluate metrics to reflect this goal.

使用现有多任务模型评估并结合模型仿真方法与个体标注者观点来预测个体标注者的不一致性，得出具体少数观点的模型更适用于模型预测。