Modern training strategies of deep neural networks (NNs) tend to induce a
heavy-tailed (HT) spectra of layer weights. Extensive efforts to study this
phenomenon have found that NNs with HT weight spectra tend to generalize well.
A prevailing notion for the occurrence of such HT spectra attributes gradient
noise during training as a key contributing factor. Our work shows that
gradient noise is unnecessary for generating HT weight spectra: two-layer NNs
trained with full-batch Gradient Descent/Adam can exhibit HT spectra in their
weights after finite training steps. To this end, we first identify the scale
of the learning rate at which one step of full-batch Adam can lead to feature
learning in the shallow NN, particularly when learning a single index teacher
model. Next, we show that multiple optimizer steps with such (sufficiently)
large learning rates can transition the bulk of the weight's spectra into an HT
distribution. To understand this behavior, we present a novel perspective based
on the singular vectors of the weight matrices and optimizer updates. We show
that the HT weight spectrum originates from the `spike', which is generated
from feature learning and interacts with the main bulk to generate an HT
spectrum. Finally, we analyze the correlations between the HT weight spectra
and generalization after multiple optimizer updates with varying learning
rates.

通过分析深度神经网络的重尾谱、梯度噪声和学习率等因素，我们揭示了深度神经网络在有限的训练步骤后可以生成重尾谱的机制，并研究了重尾谱与泛化能力之间的相关性。