Crowdsourcing platforms are often used to collect datasets for training
machine learning models, despite higher levels of inaccurate labeling compared
to expert labeling. There are two common strategies to manage the impact of
such noise. The first involves aggregating redundant annotations, but comes at
the expense of labeling substantially fewer examples. Secondly, prior works
have also considered using the entire annotation budget to label as many
examples as possible and subsequently apply denoising algorithms to implicitly
clean the dataset. We find a middle ground and propose an approach which
reserves a fraction of annotations to explicitly clean up highly probable error
samples to optimize the annotation process. In particular, we allocate a large
portion of the labeling budget to form an initial dataset used to train a
model. This model is then used to identify specific examples that appear most
likely to be incorrect, which we spend the remaining budget to relabel.
Experiments across three model variations and four natural language processing
tasks show our approach outperforms or matches both label aggregation and
advanced denoising methods designed to handle noisy labels when allocated the
same finite annotation budget.

本文提出了一种通过在注释过程中专门清理高概率错误样本的方法，以优化注释过程，并在三种模型变化和四个自然语言处理任务上进行的实验证明，该方法在分配相同的有限注释预算时优于聚合注释和高级去噪方法。