When random label noise is added to a training dataset, the prediction error of a neural network on a label-noise-free test dataset initially improves during early training but eventually deteriorates, following a U-shaped dependence on training time. This behaviour is believed to be a result of neural networks learning the pattern of clean data first and fitting the noise later in the training, a phenomenon that we refer to as clean-priority learning. In this study, we aim to explore the learning dynamics underlying this phenomenon. We theoretically demonstrate that, in the early stage of training, the update direction of gradient descent is determined by the clean subset of training data, leaving the noisy subset has minimal to no impact, resulting in a prioritization of clean learning. Moreover, we show both theoretically and experimentally, as the clean-priority learning goes on, the dominance of the gradients of clean samples over those of noisy samples diminishes, and finally results in a termination of the clean-priority learning and fitting of the noisy samples.

当训练数据集中加入随机标签噪声时，神经网络在无噪声测试数据集上的预测错误率在早期训练时会先改善后恶化，表现出U形依赖于训练时间。我们的研究旨在探究这一现象背后的学习动态。我们理论上证明了，训练早期，梯度下降的更新方向由干净训练数据集确定，噪声样本集对其影响极小，导致了对干净数据的优先学习。此外，我们理论上和实验上表明，随着清洁优先学习的进行，干净样本的梯度优势逐渐减弱，最终导致清洁优先学习的终止和对噪声样本的拟合。

早停止神经网络中清洁优先学习的出现