Data annotation remains the sine qua non of machine learning and AI. Recent
empirical work on data annotation has begun to highlight the importance of
rater diversity for fairness, model performance, and new lines of research have
begun to examine the working conditions for data annotation workers, the
impacts and role of annotator subjectivity on labels, and the potential
psychological harms from aspects of annotation work. This paper outlines a
critical genealogy of data annotation; starting with its psychological and
perceptual aspects. We draw on similarities with critiques of the rise of
computerized lab-based psychological experiments in the 1970's which question
whether these experiments permit the generalization of results beyond the
laboratory settings within which these results are typically obtained. Do data
annotations permit the generalization of results beyond the settings, or
locations, in which they were obtained? Psychology is overly reliant on
participants from Western, Educated, Industrialized, Rich, and Democratic
societies (WEIRD). Many of the people who work as data annotation platform
workers, however, are not from WEIRD countries; most data annotation workers
are based in Global South countries. Social categorizations and classifications
from WEIRD countries are imposed on non-WEIRD annotators through instructions
and tasks, and through them, on data, which is then used to train or evaluate
AI models in WEIRD countries. We synthesize evidence from several recent lines
of research and argue that data annotation is a form of automated social
categorization that risks entrenching outdated and static social categories
that are in reality dynamic and changing. We propose a framework for
understanding the interplay of the global social conditions of data annotation
with the subjective phenomenological experience of data annotation work.

数据标注是机器学习和人工智能的必要条件。最近的实证工作开始强调评分者的多样性对公平性和模型性能的重要性，并开始研究数据标注工作者的工作条件、标注人主观性对标签的影响以及标注工作的潜在心理危害。本文概述了数据标注的关键历史，并结合对上世纪 70 年代计算机实验心理学批判的相似性，质疑这些实验是否能够将结果推广到实验室以外的环境。我们指出心理学过于依赖西方、受过教育、工业化、富裕和民主社会（WEIRD）的参与者。然而，许多数据标注平台工作者并不来自 WEIRD 国家，而是来自全球南方国家。非 WEIRD 的标注人通过指导和任务受到 WEIRD 国家的社会分类和分类的强加，并通过这些分类对数据进行使用，在 WEIRD 国家中用于训练或评估人工智能模型。我们综合了几个最近的研究线索的证据，认为数据标注是一种自动化的社会分类，可能会使过时和静态的社会分类得到巩固，而这些分类在现实中是动态和变化的。我们提出了一个框架，以理解数据标注的全球社会条件与数据标注工作的主观现象之间的相互作用。