Human annotated data plays a crucial role in machine learning (ML) research
and development. However, the ethical considerations around the processes and
decisions that go into dataset annotation have not received nearly enough
attention. In this paper, we survey an array of literature that provides
insights into ethical considerations around crowdsourced dataset annotation. We
synthesize these insights, and lay out the challenges in this space along two
layers: (1) who the annotator is, and how the annotators' lived experiences can
impact their annotations, and (2) the relationship between the annotators and
the crowdsourcing platforms, and what that relationship affords them. Finally,
we introduce a novel framework, CrowdWorkSheets, for dataset developers to
facilitate transparent documentation of key decisions points at various stages
of the data annotation pipeline: task formulation, selection of annotators,
platform and infrastructure choices, dataset analysis and evaluation, and
dataset release and maintenance.

本文调查了围绕众包数据集注释的伦理问题，深入探讨了注解者的背景对其注释的影响以及注解者与众包平台的关系所带来的影响，并提出了一种透明化关键决策点的数据注释框架 ——CrowdWorkSheets。