In this work, we reveal a strong implicit bias of stochastic gradient descent
(SGD) that drives overly expressive networks to much simpler subnetworks,
thereby dramatically reducing the number of independent parameters, and
improving generalization. To reveal this bias, we identify invariant sets, or
subsets of parameter space that remain unmodified by SGD. We focus on two
classes of invariant sets that correspond to simpler subnetworks and commonly
appear in modern architectures. Our analysis uncovers that SGD exhibits a
property of stochastic attractivity towards these simpler invariant sets. We
establish a sufficient condition for stochastic attractivity based on a
competition between the loss landscape's curvature around the invariant set and
the noise introduced by stochastic gradients. Remarkably, we find that an
increased level of noise strengthens attractivity, leading to the emergence of
attractive invariant sets associated with saddle-points or local maxima of the
train loss. We observe empirically the existence of attractive invariant sets
in trained deep neural networks, implying that SGD dynamics often collapses to
simple subnetworks with either vanishing or redundant neurons. We further
demonstrate how this simplifying process of stochastic collapse benefits
generalization in a linear teacher-student framework. Finally, through this
analysis, we mechanistically explain why early training with large learning
rates for extended periods benefits subsequent generalization.

本研究揭示了 SGD 存在的强烈隐式偏差，由此驱使过度表达的神经网络倾向于变得更简单，从而显著减少独立参数数量，并改进了泛化能力。