Errors in labels obtained via human annotation adversely affect a model's
performance. Existing approaches propose ways to mitigate the effect of label
error on a model's downstream accuracy, yet little is known about its impact on
a model's disparity metrics. Here we study the effect of label error on a
model's disparity metrics. We empirically characterize how varying levels of
label error, in both training and test data, affect these disparity metrics. We
find that group calibration and other metrics are sensitive to train-time and
test-time label error -- particularly for minority groups. This disparate
effect persists even for models trained with noise-aware algorithms. To
mitigate the impact of training-time label error, we present an approach to
estimate the influence of a training input's label on a model's group disparity
metric. We empirically assess the proposed approach on a variety of datasets
and find significant improvement, compared to alternative approaches, in
identifying training inputs that improve a model's disparity metric. We
complement the approach with an automatic relabel-and-finetune scheme that
produces updated models with, provably, improved group calibration error.

标签错误对模型的不平衡指标产生显著影响，尤其是对少数群体，本文通过实证研究了训练时间和测试时间的标签错误如何影响模型的不平衡指标，并提出了一种估算训练输入标签对模型群体差异指标影响的方法，实证评估结果显示该方法能够显著改善模型的不平衡指标，并结合自动重新标记和微调策略，可以产生具有明显改善群体校准误差的更新模型。

量化和减少标签错误对模型差异度量的影响

Quantifying and mitigating the impact of label errors on model disparity  metrics

Label error is a ubiquitous problem in annotated data. Large amounts of label
error substantially degrades the quality of deep learning models. Existing
methods to tackle the label error problem largely focus on the classification
task, and either rely on task specific architecture or require non-trivial
additional computations, which is undesirable or even unattainable for industry
usage. In this paper, we propose LEDO: a model-agnostic and computationally
efficient framework for Label Error Detection and Overwrite. LEDO is based on
Monte Carlo Dropout combined with uncertainty metrics, and can be easily
generalized to multiple tasks and data sets. Applying LEDO to an industry
opinion-based question answering system demonstrates it is effective at
improving accuracy in all the core models. Specifically, LEDO brings 1.1% MRR
gain for the retrieval model, 1.5% PR AUC improvement for the machine reading
comprehension model, and 0.9% rise in the Average Precision for the ranker, on
top of the strong baselines with a large-scale social media dataset.
Importantly, LEDO is computationally efficient compared to methods that require
loss function change, and cost-effective as the resulting data can be used in
the same continuous training pipeline for production. Further analysis shows
that these gains come from an improved decision boundary after cleaning the
label errors existed in the training data.

本文提出了一个模型无关且计算高效的标签错误检测和覆盖框架 ——LEDO，基于 Monte Carlo Dropout 结合不确定性测度，并针对工业问题回答系统进行了应用，它能显著提高模型的准确性并降低成本。