Neural networks in the lazy training regime converge to kernel machines. Can
neural networks in the rich feature learning regime learn a kernel machine with
a data-dependent kernel? We demonstrate that this can indeed happen due to a
phenomenon we term silent alignment, which requires that the tangent kernel of
a network evolves in eigenstructure while small and before the loss appreciably
decreases, and grows only in overall scale afterwards. We show that such an
effect takes place in homogenous neural networks with small initialization and
whitened data. We provide an analytical treatment of this effect in the linear
network case. In general, we find that the kernel develops a low-rank
contribution in the early phase of training, and then evolves in overall scale,
yielding a function equivalent to a kernel regression solution with the final
network's tangent kernel. The early spectral learning of the kernel depends on
the depth. We also demonstrate that non-whitened data can weaken the silent
alignment effect.

本文探讨了神经网络在富特征学习阶段是否能够通过数据相关的核来学习一个核机器问题，发现这可以通过一种我们称之为 ' 静默对齐（silent alignment）' 的现象来实现，该现象要求网络的切向核在小范围内而且损失函数减少之前已经在特征上对齐，然后在总体规模上才能增长，总体而言，我们的研究表明以这种方式训练的神经网络会先学习一个低秩贡献的核，然后再总体上演化，从而得到了一个与其相当的核回归解决方案。