We show that large pre-trained language models are inherently highly capable
of identifying label errors in natural language datasets: simply examining
out-of-sample data points in descending order of fine-tuned task loss
significantly outperforms more complex error-detection mechanisms proposed in
previous work.
To this end, we contribute a novel method for introducing realistic,
human-originated label noise into existing crowdsourced datasets such as SNLI
and TweetNLP. We show that this noise has similar properties to real,
hand-verified label errors, and is harder to detect than existing synthetic
noise, creating challenges for model robustness. We argue that human-originated
noise is a better standard for evaluation than synthetic noise.
Finally, we use crowdsourced verification to evaluate the detection of real
errors on IMDB, Amazon Reviews, and Recon, and confirm that pre-trained models
perform at a 9-36% higher absolute Area Under the Precision-Recall Curve than
existing models.

该研究表明，大型预训练语言模型本质上具有高度识别自然语言数据集中标签错误的能力：仅通过按微调任务损失的降序检查样本数据点，可显著优于先前工作中提出的更复杂的错误检测机制。此外，研究对引入 SNLI 和 TweetNLP 等现有众包数据集中真实的、人工标记噪声提出了一种新方法，证明该噪声具有类似于真实手动验标错误的属性，并且比现有的合成噪声更难以检测，因此将人工起源噪声作为评估标准更好。最后，使用众包验证评估在 IMDB、Amazon 评论和 Recon 中实际错误的检测，并确认预训练模型的绝对精度召回曲线下面积比现有模型高 9-36%。

利用预训练语言模型检测标签错误

Detecting Label Errors by using Pre-Trained Language Models

We consider the problem of making machine translation more robust to
character-level variation at the source side, such as typos. Existing methods
achieve greater coverage by applying subword models such as byte-pair encoding
(BPE) and character-level encoders, but these methods are highly sensitive to
spelling mistakes. We show how training on a mild amount of random synthetic
noise can dramatically improve robustness to these variations, without
diminishing performance on clean text. We focus on translation performance on
natural noise, as captured by frequent corrections in Wikipedia edit logs, and
show that robustness to such noise can be achieved using a balanced diet of
simple synthetic noises at training time, without access to the natural noise
data or distribution.

本文提出通过引入随机合成的少量噪音来改善机器翻译中源文本字符级别波动的鲁棒性，从而提高对拼写错误等变异的覆盖率。作者表明，通过在训练时使用一些较为简单的合成噪音，可以实现对常见噪音（例如 Wikipedia 编辑日志中的修正）的鲁棒性，同时不会降低在规范文本上的表现。