We show that large pre-trained language models are inherently highly capable
of identifying label errors in natural language datasets: simply examining
out-of-sample data points in descending order of fine-tuned task loss
significantly outperforms more complex error-detection mechanisms proposed in
previous work.
To this end, we contribute a novel method for introducing realistic,
human-originated label noise into existing crowdsourced datasets such as SNLI
and TweetNLP. We show that this noise has similar properties to real,
hand-verified label errors, and is harder to detect than existing synthetic
noise, creating challenges for model robustness. We argue that human-originated
noise is a better standard for evaluation than synthetic noise.
Finally, we use crowdsourced verification to evaluate the detection of real
errors on IMDB, Amazon Reviews, and Recon, and confirm that pre-trained models
perform at a 9-36% higher absolute Area Under the Precision-Recall Curve than
existing models.

该研究表明，大型预训练语言模型本质上具有高度识别自然语言数据集中标签错误的能力：仅通过按微调任务损失的降序检查样本数据点，可显著优于先前工作中提出的更复杂的错误检测机制。此外，研究对引入 SNLI 和 TweetNLP 等现有众包数据集中真实的、人工标记噪声提出了一种新方法，证明该噪声具有类似于真实手动验标错误的属性，并且比现有的合成噪声更难以检测，因此将人工起源噪声作为评估标准更好。最后，使用众包验证评估在 IMDB、Amazon 评论和 Recon 中实际错误的检测，并确认预训练模型的绝对精度召回曲线下面积比现有模型高 9-36%。