The evolution of a deep neural network trained by the gradient descent can be
described by its neural tangent kernel (NTK) as introduced in [20], where it
was proven that in the infinite width limit the NTK converges to an explicit
limiting kernel and it stays constant during training. The NTK was also
implicit in some other recent papers [6,13,14]. In the overparametrization
regime, a fully-trained deep neural network is indeed equivalent to the kernel
regression predictor using the limiting NTK. And the gradient descent achieves
zero training loss for a deep overparameterized neural network. However, it was
observed in [5] that there is a performance gap between the kernel regression
using the limiting NTK and the deep neural networks. This performance gap is
likely to originate from the change of the NTK along training due to the finite
width effect. The change of the NTK along the training is central to describe
the generalization features of deep neural networks.
In the current paper, we study the dynamic of the NTK for finite width deep
fully-connected neural networks. We derive an infinite hierarchy of ordinary
differential equations, the neural tangent hierarchy (NTH) which captures the
gradient descent dynamic of the deep neural network. Moreover, under certain
conditions on the neural network width and the data set dimension, we prove
that the truncated hierarchy of NTH approximates the dynamic of the NTK up to
arbitrary precision. This description makes it possible to directly study the
change of the NTK for deep neural networks, and sheds light on the observation
that deep neural networks outperform kernel regressions using the corresponding
limiting NTK.

本文研究了有限宽度的深度全连接神经网络中神经切向核的动态，并推导出一个无穷层次的普通微分方程组，它捕捉了深层神经网络的梯度下降动态。此外，在条件限制下，研究证明了 NTH 的截断层次近似于 NTK 的动态。这些描述使直接研究深度神经网络的 NTK 的变化成为可能，同时也揭示了深度神经网络胜过相应极限 NTK 的内在原因。