In the presence of noisy or incorrect labels, neural networks have the undesirable tendency to memorize information about the noise. Standard regularization techniques such as dropout, weight decay or data augmentation sometimes help, but do not prevent this behavior. If one considers neural network weights as random variables that depend on the data and stochasticity of training, the amount of memorized information can be quantified with the Shannon mutual information between weights and the vector of all training labels given inputs, $I(w : \mathbf{y} \mid \mathbf{x})$. We show that for any training algorithm, low values of this term correspond to reduction in memorization of label-noise and better generalization bounds. To obtain these low values, we propose training algorithms that employ an auxiliary network that predicts gradients in the final layers of a classifier without accessing labels. We illustrate the effectiveness of our approach on versions of MNIST, CIFAR-10, and CIFAR-100 corrupted with various noise models, and on a large-scale dataset Clothing1M that has noisy labels.

研究发现，神经网络在存在噪声或不正确标签的情况下，往往会记住有关噪声的信息。为了减少这种记忆现象，提高泛化能力，本文提出使用一个辅助网络来训练，并利用Shannon互信息量化记忆的信息量。在MNIST、CIFAR-10、CIFAR-100和Clothing1M等数据集上验证了该算法的有效性。

通过控制神经网络权重中的标签噪声信息来提高泛化能力