Label corruption, where training samples have incorrect labels, can
significantly degrade the performance of machine learning models. This
corruption often arises from non-expert labeling or adversarial attacks.
Acquiring large, perfectly labeled datasets is costly, and retraining large
models from scratch when a clean dataset becomes available is computationally
expensive. To address this challenge, we propose Post-Training Correction, a
new paradigm that adjusts model parameters after initial training to mitigate
label noise, eliminating the need for retraining. We introduce Verifix, a novel
Singular Value Decomposition (SVD) based algorithm that leverages a small,
verified dataset to correct the model weights using a single update. Verifix
uses SVD to estimate a Clean Activation Space and then projects the model's
weights onto this space to suppress activations corresponding to corrupted
data. We demonstrate Verifix's effectiveness on both synthetic and real-world
label noise. Experiments on the CIFAR dataset with 25% synthetic corruption
show 7.36% generalization improvements on average. Additionally, we observe
generalization improvements of up to 2.63% on naturally corrupted datasets like
WebVision1.0 and Clothing1M.

提出后训练校正新范式以解决标签污染问题，引入基于奇异值分解（Singular Value Decomposition，SVD）的 Verifix 算法以消除机器学习模型中的错误标签，并通过投影模型权重到清洁激活空间来抑制对应于损坏数据的激活，实验证明 Verifix 在合成和真实数据集上都能有效提升泛化性能。

Verifix: 通过验证样本改善标签噪声鲁棒性的后训练纠正

Verifix: Post-Training Correction to Improve Label Noise Robustness with  Verified Samples

Self-distillation (SD) is the process of first training a \enquote{teacher}
model and then using its predictions to train a \enquote{student} model with
the \textit{same} architecture. Specifically, the student's objective function
is $\big(\xi*\ell(\text{teacher's predictions}, \text{ student's predictions})
+ (1-\xi)*\ell(\text{given labels}, \text{ student's predictions})\big)$, where
$\ell$ is some loss function and $\xi$ is some parameter $\in [0,1]$.
Empirically, SD has been observed to provide performance gains in several
settings. In this paper, we theoretically characterize the effect of SD in two
supervised learning problems with \textit{noisy labels}. We first analyze SD
for regularized linear regression and show that in the high label noise regime,
the optimal value of $\xi$ that minimizes the expected error in estimating the
ground truth parameter is surprisingly greater than 1. Empirically, we show
that $\xi > 1$ works better than $\xi \leq 1$ even with the cross-entropy loss
for several classification datasets when 50\% or 30\% of the labels are
corrupted. Further, we quantify when optimal SD is better than optimal
regularization. Next, we analyze SD in the case of logistic regression for
binary classification with random label corruption and quantify the range of
label corruption in which the student outperforms the teacher in terms of
accuracy. To our knowledge, this is the first result of its kind for the
cross-entropy loss.

研究了在带有噪声标签的监督学习问题中，使用 SD 的效果，并在理论上和经验上表明，最优的 SD 参数在某些情况下大于 1，证明 SD 比优化有用，并在二元分类和随机标签损坏的情况下证明了学生比教师有更好的准确性。

在标签噪声存在的情况下理解自蒸馏

Understanding Self-Distillation in the Presence of Label Noise

Self-supervision provides effective representations for downstream tasks
without requiring labels. However, existing approaches lag behind fully
supervised training and are often not thought beneficial beyond obviating or
reducing the need for annotations. We find that self-supervision can benefit
robustness in a variety of ways, including robustness to adversarial examples,
label corruption, and common input corruptions. Additionally, self-supervision
greatly benefits out-of-distribution detection on difficult, near-distribution
outliers, so much so that it exceeds the performance of fully supervised
methods. These results demonstrate the promise of self-supervision for
improving robustness and uncertainty estimation and establish these tasks as
new axes of evaluation for future self-supervised learning research.

自我监督能够提高深度学习在鲁棒性、异常检测等难题上的表现甚至超越了全监督学习，成为未来自我监督学习研究的新方向。