Neural Tangent Kernel (NTK) is widely used to analyze overparametrized neural
networks due to the famous result by Jacot et al. (2018): in the infinite-width
limit, the NTK is deterministic and constant during training. However, this
result cannot explain the behavior of deep networks, since it generally does
not hold if depth and width tend to infinity simultaneously. In this paper, we
study the NTK of fully-connected ReLU networks with depth comparable to width.
We prove that the NTK properties depend significantly on the depth-to-width
ratio and the distribution of parameters at initialization. In fact, our
results indicate the importance of the three phases in the hyperparameter space
identified in Poole et al. (2016): ordered, chaotic and the edge of chaos
(EOC). We derive exact expressions for the NTK dispersion in the
infinite-depth-and-width limit in all three phases and conclude that the NTK
variability grows exponentially with depth at the EOC and in the chaotic phase
but not in the ordered phase. We also show that the NTK of deep networks may
stay constant during training only in the ordered phase and discuss how the
structure of the NTK matrix changes during training.

本文研究了深度与宽度相当的全连接 ReLU 网络的神经切向核（Neural Tangent Kernel）及其性质，发现其性质取决于深度与宽度之比以及初始状态下参数分布的情况。结果表明，在超参数空间中，有序、混沌和混沌边缘三个阶段很重要。在混沌和混沌边缘阶段，NTK 可变性随着深度呈指数增长，但在有序阶段则不会，此外还展示了深度神经网络的 NTK 只有在有序阶段中才能在训练过程中保持恒定，并探讨了 NTK 矩阵在训练过程中的结构变化。