Data quality is crucial for training accurate, unbiased, and trustworthy
machine learning models and their correct evaluation. Recent works, however,
have shown that even popular datasets used to train and evaluate
state-of-the-art models contain a non-negligible amount of erroneous
annotations, bias or annotation artifacts. There exist best practices and
guidelines regarding annotation projects. But to the best of our knowledge, no
large-scale analysis has been performed as of yet on how quality management is
actually conducted when creating natural language datasets and whether these
recommendations are followed. Therefore, we first survey and summarize
recommended quality management practices for dataset creation as described in
the literature and provide suggestions on how to apply them. Then, we compile a
corpus of 591 scientific publications introducing text datasets and annotate it
for quality-related aspects, such as annotator management, agreement,
adjudication or data validation. Using these annotations, we then analyze how
quality management is conducted in practice. We find that a majority of the
annotated publications apply good or very good quality management. However, we
deem the effort of 30% of the works as only subpar. Our analysis also shows
common errors, especially with using inter-annotator agreement and computing
annotation error rates.

文章调研了自然语言数据集创作的质量管理实践，分析了 591 个科学出版物并发现了其普遍存在的错误和 30% 的工作只达到了次级水平，尤其在使用标注错误率和计算注释错误率方面存在常见错误。

野外数据集标注质量管理分析

Analyzing Dataset Annotation Quality Management in the Wild

Linguistically diverse datasets are critical for training and evaluating
robust machine learning systems, but data collection is a costly process that
often requires experts. Crowdsourcing the process of paraphrase generation is
an effective means of expanding natural language datasets, but there has been
limited analysis of the trade-offs that arise when designing tasks. In this
paper, we present the first systematic study of the key factors in
crowdsourcing paraphrase collection. We consider variations in instructions,
incentives, data domains, and workflows. We manually analyzed paraphrases for
correctness, grammaticality, and linguistic diversity. Our observations provide
new insight into the trade-offs between accuracy and diversity in crowd
responses that arise as a result of task design, providing guidance for future
paraphrase generation procedures.

本研究是对众包方法进行语义重述数据采集的系统研究，旨在探讨任务设计中准确性和多样性之间的平衡关系，并提出未来语义重述集采集的指导意见。