Design biases in NLP systems, such as performance differences for different
populations, often stem from their creator's positionality, i.e., views and
lived experiences shaped by identity and background. Despite the prevalence and
risks of design biases, they are hard to quantify because researcher, system,
and dataset positionality is often unobserved. We introduce NLPositionality, a
framework for characterizing design biases and quantifying the positionality of
NLP datasets and models. Our framework continuously collects annotations from a
diverse pool of volunteer participants on LabintheWild, and statistically
quantifies alignment with dataset labels and model predictions. We apply
NLPositionality to existing datasets and models for two tasks -- social
acceptability and hate speech detection. To date, we have collected 16,299
annotations in over a year for 600 instances from 1,096 annotators across 87
countries. We find that datasets and models align predominantly with Western,
White, college-educated, and younger populations. Additionally, certain groups,
such as non-binary people and non-native English speakers, are further
marginalized by datasets and models as they rank least in alignment across all
tasks. Finally, we draw from prior literature to discuss how researchers can
examine their own positionality and that of their datasets and models, opening
the door for more inclusive NLP systems.

NLPositionality 框架用于检测自然语言处理（NLP）模型及数据集的偏差和位置特征，研究发现现有的数据集和模型普遍偏向西方白人、受过高等教育的年轻人，并排斥一些群体如非二元人群和非英语母语者，本文提出如何检查 NLP 模型及数据集的创造者的位置特征。