At initialization, artificial neural networks (ANNs) are equivalent to Gaussian processes in the infinite-width limit, thus connecting them to kernel methods. We prove that the evolution of an ANN during training can also be described by a kernel: during gradient descent on the parameters of an ANN, the network function $f_\theta$ (which maps input vectors to output vectors) follows the kernel gradient of the functional cost (which is convex, in contrast to the parameter cost) w.r.t. a new kernel: the Neural Tangent Kernel (NTK). This kernel is central to describe the generalization features of ANNs. While the NTK is random at initialization and varies during training, in the infinite-width limit it converges to an explicit limiting kernel and it stays constant during training. This makes it possible to study the training of ANNs in function space instead of parameter space. Convergence of the training can then be related to the positive-definiteness of the limiting NTK. We prove the positive-definiteness of the limiting NTK when the data is supported on the sphere and the non-linearity is non-polynomial. We then focus on the setting of least-squares regression and show that in the infinite-width limit, the network function $f_\theta$ follows a linear differential equation during training. The convergence is fastest along the largest kernel principal components of the input data with respect to the NTK, hence suggesting a theoretical motivation for early stopping. Finally we study the NTK numerically, observe its behavior for wide networks, and compare it to the infinite-width limit.

本研究证明了在梯度下降算法中，人工神经网络的演化可以被表示为一种核函数，称为神经切向核。它在无限宽度下收敛于一个明确的极限核，并且在训练过程中保持不变，可以用函数空间而不是参数空间来研究人工神经网络的训练。我们关注最小二乘回归并表明，在无限宽度下，网络函数$f_	heta$在训练期间遵循线性微分方程。最后，我们对神经切向核进行了数值研究，观察了其在宽网络中的行为，并将其与无限宽度的极限进行了比较。

神经正切核:神经网络的收敛性和泛化性