Given data with noisy labels, over-parameterized deep networks suffer
overfitting mislabeled data, resulting in poor generalization. The memorization
effect of deep networks shows that although the networks have the ability to
memorize all noisy data, they would first memorize clean training data, and
then gradually memorize mislabeled training data. A simple and effective method
that exploits the memorization effect to combat noisy labels is early stopping.
However, early stopping cannot distinguish the memorization of clean data and
mislabeled data, resulting in the network still inevitably overfitting
mislabeled data in the early training stage.In this paper, to decouple the
memorization of clean data and mislabeled data, and further reduce the side
effect of mislabeled data, we perform additive decomposition on network
parameters. Namely, all parameters are additively decomposed into two groups,
i.e., parameters $\mathbf{w}$ are decomposed as
$\mathbf{w}=\bm{\sigma}+\bm{\gamma}$. Afterward, the parameters $\bm{\sigma}$
are considered to memorize clean data, while the parameters $\bm{\gamma}$ are
considered to memorize mislabeled data. Benefiting from the memorization
effect, the updates of the parameters $\bm{\sigma}$ are encouraged to fully
memorize clean data in early training, and then discouraged with the increase
of training epochs to reduce interference of mislabeled data. The updates of
the parameters $\bm{\gamma}$ are the opposite. In testing, only the parameters
$\bm{\sigma}$ are employed to enhance generalization. Extensive experiments on
both simulated and real-world benchmarks confirm the superior performance of
our method.

通过参数的加性分解和早停策略，本研究在深度网络中利用记忆效应来对抗有噪声的标签数据，并成功地区分了对干净数据和错标数据的记忆，从而减少错标数据的干扰并提高了推广性能。

利用网络参数叠加分解解决噪声标签问题

Tackling Noisy Labels with Network Parameter Additive Decomposition

Score-based Generative Models (SGMs) is one leading method in generative
modeling, renowned for their ability to generate high-quality samples from
complex, high-dimensional data distributions. The method enjoys empirical
success and is supported by rigorous theoretical convergence properties. In
particular, it has been shown that SGMs can generate samples from a
distribution that is close to the ground-truth if the underlying score function
is learned well, suggesting the success of SGM as a generative model. We
provide a counter-example in this paper. Through the sample complexity
argument, we provide one specific setting where the score function is learned
well. Yet, SGMs in this setting can only output samples that are Gaussian
blurring of training data points, mimicking the effects of kernel density
estimation. The finding resonates a series of recent finding that reveal that
SGMs can demonstrate strong memorization effect and fail to generate.

通过样本复杂性论证，我们在特定设置下展示了得分函数训练良好的情况下，SGM 只能输出训练数据点的高斯模糊样本，从而模拟核密度估计的效果，这与最近的研究结果一致，揭示了 SGM 展示出记忆效应并且无法生成的弱点。

得分高不代表生成模型好

A Good Score Does not Lead to A Good Generative Model

Weakly supervised multi-label classification (WSML) task, which is to learn a
multi-label classification using partially observed labels per image, is
becoming increasingly important due to its huge annotation cost. In this work,
we first regard unobserved labels as negative labels, casting the WSML task
into noisy multi-label classification. From this point of view, we empirically
observe that memorization effect, which was first discovered in a noisy
multi-class setting, also occurs in a multi-label setting. That is, the model
first learns the representation of clean labels, and then starts memorizing
noisy labels. Based on this finding, we propose novel methods for WSML which
reject or correct the large loss samples to prevent model from memorizing the
noisy label. Without heavy and complex components, our proposed methods
outperform previous state-of-the-art WSML methods on several partial label
settings including Pascal VOC 2012, MS COCO, NUSWIDE, CUB, and OpenImages V3
datasets. Various analysis also show that our methodology actually works well,
validating that treating large loss properly matters in a weakly supervised
multi-label classification. Our code is available at
this https URL

本文探究弱监督多标签分类任务中的噪声与过拟合问题，并提出了能够处理大损失值样本的方法，得出准确性更高的预测结果。