We study when the neural tangent kernel (NTK) approximation is valid for
training a model with the square loss. In the lazy training setting of Chizat
et al. 2019, we show that rescaling the model by a factor of $\alpha = O(T)$
suffices for the NTK approximation to be valid until training time $T$. Our
bound is tight and improves on the previous bound of Chizat et al. 2019, which
required a larger rescaling factor of $\alpha = O(T^2)$.

本篇研究探讨了神经切线核 (NTK) 近似在用方形误差训练模型中的有效性。在 Chizat 等人 2019 年的懒惰训练设置中，我们证明通过 O (T) 的模型缩放因子使得 NTK 近似能够有效直到训练时间 T。我们的界限紧密且优于 Chizat 等人 2019 年的先前界限，该界限需要更大的缩放因子 O (T^2)。

NTK 逼近在更长时间内有效

The NTK approximation is valid for longer than you think

Two distinct limits for deep learning have been derived as the network width
$h\rightarrow \infty$, depending on how the weights of the last layer scale
with $h$. In the Neural Tangent Kernel (NTK) limit, the dynamics becomes linear
in the weights and is described by a frozen kernel $\Theta$. By contrast, in
the Mean-Field limit, the dynamics can be expressed in terms of the
distribution of the parameters associated with a neuron, that follows a partial
differential equation. In this work we consider deep networks where the weights
in the last layer scale as $\alpha h^{-1/2}$ at initialization. By varying
$\alpha$ and $h$, we probe the crossover between the two limits. We observe the
previously identified regimes of lazy training and feature training. In the
lazy-training regime, the dynamics is almost linear and the NTK barely changes
after initialization. The feature-training regime includes the mean-field
formulation as a limiting case and is characterized by a kernel that evolves in
time, and learns some features. We perform numerical experiments on MNIST,
Fashion-MNIST, EMNIST and CIFAR10 and consider various architectures. We find
that (i) The two regimes are separated by an $\alpha^*$ that scales as
$h^{-1/2}$. (ii) Network architecture and data structure play an important role
in determining which regime is better: in our tests, fully-connected networks
perform generally better in the lazy-training regime, unlike convolutional
networks. (iii) In both regimes, the fluctuations $\delta F$ induced on the
learned function by initial conditions decay as $\delta F\sim 1/\sqrt{h}$,
leading to a performance that increases with $h$. The same improvement can also
be obtained at an intermediate width by ensemble-averaging several networks.
(iv) In the feature-training regime we identify a time scale
$t_1\sim\sqrt{h}\alpha$, such that for $t\ll t_1$ the dynamics is linear.

该论文针对深度学习的 Neural Tangent Kernel 极限和 Mean-Field 极限进行了研究，发现不同的调参可以使得网络在 lazy training 和 feature training 两种状态下表现不同，并提出了一种中间状态下集合平均方法可以提高性能。

深度神经网络中特征和懒惰训练的解耦

Disentangling feature and lazy training in deep neural networks

In a series of recent theoretical works, it was shown that strongly
over-parameterized neural networks trained with gradient-based methods could
converge exponentially fast to zero training loss, with their parameters hardly
varying. In this work, we show that this "lazy training" phenomenon is not
specific to over-parameterized neural networks, and is due to a choice of
scaling, often implicit, that makes the model behave as its linearization
around the initialization, thus yielding a model equivalent to learning with
positive-definite kernels. Through a theoretical analysis, we exhibit various
situations where this phenomenon arises in non-convex optimization and we
provide bounds on the distance between the lazy and linearized optimization
paths. Our numerical experiments bring a critical note, as we observe that the
performance of commonly used non-linear deep convolutional neural networks in
computer vision degrades when trained in the lazy regime. This makes it
unlikely that "lazy training" is behind the many successes of neural networks
in difficult high dimensional tasks.

本文通过理论分析和数值实验，证明了惰性训练现象不仅仅局限于过度参数化的神经网络，同时也适用于其他优化问题，然而，在计算机视觉中，惰性训练并不适用于深度卷积神经网络的训练，这使得惰性训练很可能不是神经网络在高维任务中成功的原因之一。