Instruction-tuning has become an integral part of training pipelines for
Large Language Models (LLMs) and has been shown to yield strong performance
gains. In an orthogonal line of research, Annotation Error Detection (AED) has
emerged as a tool for detecting quality issues of gold-standard labels. But so
far, the application of AED methods is limited to discriminative settings. It
is an open question how well AED methods generalize to generative settings
which are becoming widespread via generative LLMs. In this work, we present a
first and new benchmark for AED on instruction-tuning data: Donkii. It
encompasses three instruction-tuning datasets enriched with annotations by
experts and semi-automatic methods. We find that all three datasets contain
clear-cut errors that sometimes directly propagate into instruction-tuned LLMs.
We propose four AED baselines for the generative setting and evaluate them
comprehensively on the newly introduced dataset. Our results demonstrate that
choosing the right AED method and model size is indeed crucial, thereby
deriving practical recommendations. To gain insights, we provide a first
case-study to examine how the quality of the instruction-tuning datasets
influences downstream performance.

在这项研究中，我们提出了一个新的 AED 基准测试：Donkii，它包含了三个经过专家和半自动方法注释的指导调整数据集。我们发现这三个数据集中包含明显的错误，有时直接传播到指导调整的 LLMs 中。我们提出了四个适用于生成设置的 AED 基准，并在新引入的数据集上进行了全面评估。我们的结果表明选择正确的 AED 方法和模型大小确实至关重要，从而得出了实际建议。为了获得更多见解，我们提供了第一个案例研究，以检查指导调整数据集的质量对下游性能的影响。

Donkii：指导调校数据集中的注释错误检测方法能否发现错误？

Donkii: Can Annotation Error Detection Methods Find Errors in  Instruction-Tuning Datasets?

Manually annotated datasets are crucial for training and evaluating Natural
Language Processing models. However, recent work has discovered that even
widely-used benchmark datasets contain a substantial number of erroneous
annotations. This problem has been addressed with Annotation Error Detection
(AED) models, which can flag such errors for human re-annotation. However, even
though many of these AED methods assume a final curation step in which a human
annotator decides whether the annotation is erroneous, they have been developed
as static models without any human-in-the-loop component. In this work, we
propose ActiveAED, an AED method that can detect errors more accurately by
repeatedly querying a human for error corrections in its prediction loop. We
evaluate ActiveAED on eight datasets spanning five different tasks and find
that it leads to improvements over the state of the art on seven of them, with
gains of up to six percentage points in average precision.

本文提出了 ActiveAED 的 AED 方法，可在预测循环中反复查询人类进行错误修正，从而更准确地检测错误，并在八个数据集中获得了提高的结果。

ActiveAED: 人在循环中改善注释错误检测

ActiveAED: A Human in the Loop Improves Annotation Error Detection

Annotated data is an essential ingredient in natural language processing for
training and evaluating machine learning models. It is therefore very desirable
for the annotations to be of high quality. Recent work, however, has shown that
several popular datasets contain a surprising amount of annotation errors or
inconsistencies. To alleviate this issue, many methods for annotation error
detection have been devised over the years. While researchers show that their
approaches work well on their newly introduced datasets, they rarely compare
their methods to previous work or on the same datasets. This raises strong
concerns on methods' general performance and makes it difficult to asses their
strengths and weaknesses. We therefore reimplement 18 methods for detecting
potential annotation errors and evaluate them on 9 English datasets for text
classification as well as token and span labeling. In addition, we define a
uniform evaluation setup including a new formalization of the annotation error
detection task, evaluation protocol and general best practices. To facilitate
future research and reproducibility, we release our datasets and
implementations in an easy-to-use and open source software package.

本文重新实现并评估了 18 种检测可能存在的注释错误的方法，并在 9 个英文数据集上进行了评估，同时提供了评估协议和实现的开源软件包，以促进未来的研究和再现性。