Given data with noisy labels, over-parameterized deep networks suffer
overfitting mislabeled data, resulting in poor generalization. The memorization
effect of deep networks shows that although the networks have the ability to
memorize all noisy data, they would first memorize clean training data, and
then gradually memorize mislabeled training data. A simple and effective method
that exploits the memorization effect to combat noisy labels is early stopping.
However, early stopping cannot distinguish the memorization of clean data and
mislabeled data, resulting in the network still inevitably overfitting
mislabeled data in the early training stage.In this paper, to decouple the
memorization of clean data and mislabeled data, and further reduce the side
effect of mislabeled data, we perform additive decomposition on network
parameters. Namely, all parameters are additively decomposed into two groups,
i.e., parameters $\mathbf{w}$ are decomposed as
$\mathbf{w}=\bm{\sigma}+\bm{\gamma}$. Afterward, the parameters $\bm{\sigma}$
are considered to memorize clean data, while the parameters $\bm{\gamma}$ are
considered to memorize mislabeled data. Benefiting from the memorization
effect, the updates of the parameters $\bm{\sigma}$ are encouraged to fully
memorize clean data in early training, and then discouraged with the increase
of training epochs to reduce interference of mislabeled data. The updates of
the parameters $\bm{\gamma}$ are the opposite. In testing, only the parameters
$\bm{\sigma}$ are employed to enhance generalization. Extensive experiments on
both simulated and real-world benchmarks confirm the superior performance of
our method.

通过参数的加性分解和早停策略，本研究在深度网络中利用记忆效应来对抗有噪声的标签数据，并成功地区分了对干净数据和错标数据的记忆，从而减少错标数据的干扰并提高了推广性能。

利用网络参数叠加分解解决噪声标签问题

Tackling Noisy Labels with Network Parameter Additive Decomposition

Recently, over-parameterized deep networks, with increasingly more network
parameters than training samples, have dominated the performances of modern
machine learning. However, when the training data is corrupted, it has been
well-known that over-parameterized networks tend to overfit and do not
generalize. In this work, we propose a principled approach for robust training
of over-parameterized deep networks in classification tasks where a proportion
of training labels are corrupted. The main idea is yet very simple: label noise
is sparse and incoherent with the network learned from clean data, so we model
the noise and learn to separate it from the data. Specifically, we model the
label noise via another sparse over-parameterization term, and exploit implicit
algorithmic regularizations to recover and separate the underlying corruptions.
Remarkably, when trained using such a simple method in practice, we demonstrate
state-of-the-art test accuracy against label noise on a variety of real
datasets. Furthermore, our experimental results are corroborated by theory on
simplified linear models, showing that exact separation between sparse noise
and low-rank data can be achieved under incoherent conditions. The work opens
many interesting directions for improving over-parameterized models by using
sparse over-parameterization and implicit regularization.

为解决对于干扰数据的过度拟合而影响泛化效果，我们提出了一种基于稀疏过参数化和隐式正则化的优化方案，通过对标签噪声进行建模和分离，实现了对于训练数据中的噪声数据清洗，最终获得了在多个真实数据集上较先进的测试结果，同时为优化过度参数化模型开辟了许多有趣的方向。

通过过度参数化实现标签噪音下的稳健训练

Robust Training under Label Noise by Over-parameterization

The recently proposed Lottery Ticket Hypothesis of Frankle and Carbin (2019)
suggests that the performance of over-parameterized deep networks is due to the
random initialization seeding the network with a small fraction of favorable
weights. These weights retain their dominant status throughout training -- in a
very real sense, this sub-network "won the lottery" during initialization. The
authors find sub-networks via unstructured magnitude pruning with 85-95% of
parameters removed that train to the same accuracy as the original network at a
similar speed, which they call winning tickets. In this paper, we extend the
Lottery Ticket Hypothesis to a variety of transfer learning tasks. We show that
sparse sub-networks with approximately 90-95% of weights removed achieve (and
often exceed) the accuracy of the original dense network in several realistic
settings. We experimentally validate this by transferring the sparse
representation found via pruning on CIFAR-10 to SmallNORB and FashionMNIST for
object recognition tasks.

本文基于 Lottery Ticket Hypothesis 对一系列迁移学习任务进行研究，通过使用无结构量级剪枝来发现最优子网络，实验结果表明减少约 90-95% 权重的稀疏子网络在多个现实场景下能够达到或超过原始网络的准确度。