Despite its prevalence, giving up and starting over may seem wasteful in many
situations such as searching for a target or training deep neural networks
(DNNs). Our study, though, demonstrates that restarting from a checkpoint can
significantly improve generalization performance when training DNNs with noisy
labels. In the presence of noisy labels, DNNs initially learn the general
patterns of the data but then gradually overfit to the noisy labels. To combat
this overfitting phenomenon, we developed a method based on stochastic
restarting, which has been actively explored in the statistical physics field
for finding targets efficiently. By approximating the dynamics of stochastic
gradient descent into Langevin dynamics, we theoretically show that restarting
can provide great improvements as the batch size and the proportion of
corrupted data increase. We then empirically validate our theory, confirming
the significant improvements achieved by restarting. An important aspect of our
method is its ease of implementation and compatibility with other methods,
while still yielding notably improved performance. We envision it as a valuable
tool that can complement existing methods for handling noisy labels.

重启训练在含有噪声标签的深度神经网络中显著提高泛化性能，通过近似随机梯度下降动力学为朗之万动力学，我们理论上证明了重启可以在批量大小和损坏数据比例增加时提供巨大改进。同时，我们实验证实了我们的理论，证明了通过重启可以显著改善性能，并且我们的方法容易实施且与其他方法兼容，是处理噪声标签的有价值工具。