Neural networks trained with stochastic gradient descent exhibit an inductive bias towards simpler decision boundaries, typically converging to a narrow family of functions, and often fail to capture more complex features. This phenomenon raises concerns about the capacity of deep models to adequately learn and represent real-world datasets. Traditional approaches such as explicit regularization, data augmentation, architectural modifications, etc., have largely proven ineffective in encouraging the models to learn diverse features. In this work, we investigate the impact of pre-training models with noisy labels on the dynamics of SGD across various architectures and datasets. We show that pretraining promotes learning complex functions and diverse features in the presence of noise. Our experiments demonstrate that pre-training with noisy labels encourages gradient descent to find alternate minima that do not solely depend upon simple features, rather learns more complex and broader set of features, without hurting performance.

本研究探讨了带有标签噪声的预训练模型对随机梯度下降(SGD)动态的影响，解决了深度模型在学习复杂特征时的能力不足问题。研究结果表明，预训练可以在存在噪声的情况下促进学习复杂函数和多样特征，从而提升模型的表现。通过实验证明，预训练能够帮助梯度下降找到替代最小值，使得模型学习更复杂的特征而不影响性能。

标签噪声对学习复杂特征的影响