Crowdsourcing has emerged as a popular approach for collecting annotated data
to train supervised machine learning models. However, annotator bias can lead
to defective annotations. Though there are a few works investigating individual
annotator bias, the group effects in annotators are largely overlooked. In this
work, we reveal that annotators within the same demographic group tend to show
consistent group bias in annotation tasks and thus we conduct an initial study
on annotator group bias. We first empirically verify the existence of annotator
group bias in various real-world crowdsourcing datasets. Then, we develop a
novel probabilistic graphical framework GroupAnno to capture annotator group
bias with a new extended Expectation Maximization (EM) training algorithm. We
conduct experiments on both synthetic and real-world datasets. Experimental
results demonstrate the effectiveness of our model in modeling annotator group
bias in label aggregation and model learning over competitive baselines.

这篇研究论文介绍了一种新的基于概率图形框架的群体偏差建模方法，以及用于捕捉同一人群内标注员偏倚影响标注结果的训练算法，该方法在实验证明了其在标签汇聚和模型学习方面的有效性。

众包中注释者组偏见探究

Toward Annotator Group Bias in Crowdsourcing

Crowdsourcing has been the prevalent paradigm for creating natural language
understanding datasets in recent years. A common crowdsourcing practice is to
recruit a small number of high-quality workers, and have them massively
generate examples. Having only a few workers generate the majority of examples
raises concerns about data diversity, especially when workers freely generate
sentences. In this paper, we perform a series of experiments showing these
concerns are evident in three recent NLP datasets. We show that model
performance improves when training with annotator identifiers as features, and
that models are able to recognize the most productive annotators. Moreover, we
show that often models do not generalize well to examples from annotators that
did not contribute to the training set. Our findings suggest that annotator
bias should be monitored during dataset creation, and that test set annotators
should be disjoint from training set annotators.

本文通过实验展示了人群智力数据集在自然语言理解中的存在问题和缺陷，建议在数据集创建过程中监控注释者的偏见，测试集注释者应该与训练集注释者无交集。