While human annotations play a crucial role in language technologies,
annotator subjectivity has long been overlooked in data collection. Recent
studies that have critically examined this issue are often situated in the
Western context, and solely document differences across age, gender, or racial
groups. As a result, NLP research on subjectivity have overlooked the fact that
individuals within demographic groups may hold diverse values, which can
influence their perceptions beyond their group norms. To effectively
incorporate these considerations into NLP pipelines, we need datasets with
extensive parallel annotations from various social and cultural groups. In this
paper we introduce the \dataset dataset: a large-scale cross-cultural dataset
of parallel annotations for offensive language in over 4.5K sentences annotated
by a pool of over 4k annotators, balanced across gender and age, from across 21
countries, representing eight geo-cultural regions. The dataset contains
annotators' moral values captured along six moral foundations: care, equality,
proportionality, authority, loyalty, and purity. Our analyses reveal
substantial regional variations in annotators' perceptions that are shaped by
individual moral values, offering crucial insights for building pluralistic,
culturally sensitive NLP models.

通过广泛的平行标注来考虑不同社会和文化群体的道德价值观，在跨文化数据集中揭示了标注者感知的区域差异，为构建包容性、具有文化敏感性的自然语言处理模型提供了关键见解。

D3CODE: 跨文化数据中的冒犯性检测与评估中的差异分析

D3CODE: Disentangling Disagreements in Data across Cultures on  Offensiveness Detection and Evaluation

Natural language processing research has begun to embrace the notion of
annotator subjectivity, motivated by variations in labelling. This approach
understands each annotator's view as valid, which can be highly suitable for
tasks that embed subjectivity, e.g., sentiment analysis. However, this
construction may be inappropriate for tasks such as hate speech detection, as
it affords equal validity to all positions on e.g., sexism or racism. We argue
that the conflation of hate and offence can invalidate findings on hate speech,
and call for future work to be situated in theory, disentangling hate from its
orthogonal concept, offence.

自然语言处理研究已经开始采纳主观性标注者的观点，因为标注的差异性。这种方法将每个标注者的观点视为有效的，适用于涉及主观性的任务，例如情感分析。然而，这种构建对于憎恨言论检测等任务可能不适当，因为它将性别主义或种族主义等问题上的各个立场视为同等有效。我们认为混淆仇恨和冒犯可能会使关于憎恨言论的发现无效，并呼吁未来的工作要固定在理论上，从离散思考憎恨和冒犯的概念。

主观的主义？论将仇恨与冒犯混淆在滥用语言检测中的危险

Subjective $\textit{Isms}$? On the Danger of Conflating Hate and Offence  in Abusive Language Detection

Data annotation remains the sine qua non of machine learning and AI. Recent
empirical work on data annotation has begun to highlight the importance of
rater diversity for fairness, model performance, and new lines of research have
begun to examine the working conditions for data annotation workers, the
impacts and role of annotator subjectivity on labels, and the potential
psychological harms from aspects of annotation work. This paper outlines a
critical genealogy of data annotation; starting with its psychological and
perceptual aspects. We draw on similarities with critiques of the rise of
computerized lab-based psychological experiments in the 1970's which question
whether these experiments permit the generalization of results beyond the
laboratory settings within which these results are typically obtained. Do data
annotations permit the generalization of results beyond the settings, or
locations, in which they were obtained? Psychology is overly reliant on
participants from Western, Educated, Industrialized, Rich, and Democratic
societies (WEIRD). Many of the people who work as data annotation platform
workers, however, are not from WEIRD countries; most data annotation workers
are based in Global South countries. Social categorizations and classifications
from WEIRD countries are imposed on non-WEIRD annotators through instructions
and tasks, and through them, on data, which is then used to train or evaluate
AI models in WEIRD countries. We synthesize evidence from several recent lines
of research and argue that data annotation is a form of automated social
categorization that risks entrenching outdated and static social categories
that are in reality dynamic and changing. We propose a framework for
understanding the interplay of the global social conditions of data annotation
with the subjective phenomenological experience of data annotation work.

数据标注是机器学习和人工智能的必要条件。最近的实证工作开始强调评分者的多样性对公平性和模型性能的重要性，并开始研究数据标注工作者的工作条件、标注人主观性对标签的影响以及标注工作的潜在心理危害。本文概述了数据标注的关键历史，并结合对上世纪 70 年代计算机实验心理学批判的相似性，质疑这些实验是否能够将结果推广到实验室以外的环境。我们指出心理学过于依赖西方、受过教育、工业化、富裕和民主社会（WEIRD）的参与者。然而，许多数据标注平台工作者并不来自 WEIRD 国家，而是来自全球南方国家。非 WEIRD 的标注人通过指导和任务受到 WEIRD 国家的社会分类和分类的强加，并通过这些分类对数据进行使用，在 WEIRD 国家中用于训练或评估人工智能模型。我们综合了几个最近的研究线索的证据，认为数据标注是一种自动化的社会分类，可能会使过时和静态的社会分类得到巩固，而这些分类在现实中是动态和变化的。我们提出了一个框架，以理解数据标注的全球社会条件与数据标注工作的主观现象之间的相互作用。