Recently, there has been growing evidence that if the width and depth of a
neural network are scaled toward the so-called rich feature learning limit
($\mu$P and its depth extension), then some hyperparameters - such as the
learning rate - exhibit transfer from small to very large models, thus reducing
the cost of hyperparameter tuning. From an optimization perspective, this
phenomenon is puzzling, as it implies that the loss landscape is remarkably
consistent across very different model sizes. In this work, we find empirical
evidence that learning rate transfer can be attributed to the fact that under
$\mu$P and its depth extension, the largest eigenvalue of the training loss
Hessian (i.e. the sharpness) is largely independent of the width and depth of
the network for a sustained period of training time. On the other hand, we show
that under the neural tangent kernel (NTK) regime, the sharpness exhibits very
different dynamics at different scales, thus preventing learning rate transfer.
But what causes these differences in the sharpness dynamics? Through a
connection between the spectra of the Hessian and the NTK matrix, we argue that
the cause lies in the presence (for $\mu$P) or progressive absence (for the NTK
regime) of feature learning, which results in a different evolution of the NTK,
and thus of the sharpness. We corroborate our claims with a substantial suite
of experiments, covering a wide range of datasets and architectures: from
ResNets and Vision Transformers trained on benchmark vision datasets to
Transformers-based language models trained on WikiText

最近的研究表明，神经网络的宽度和深度对于所谓的丰富特征学习极限（μP 及其深层延伸）的缩放存在转移学习现象，从而降低了超参数调整的成本。本研究通过实证找到了学习率转移的实验证据，并得出结论，在 μP 及其深层延伸下，训练损失 Hessian（即锋度）的最大特征值与网络的宽度和深度在持续训练时间内基本上是独立的。