Neural Tangent Kernel (NTK) is widely used to analyze overparametrized neural
networks due to the famous result by Jacot et al. (2018): in the infinite-width
limit, the NTK is deterministic and constant during training. However, this
result cannot explain the behavior of deep networks, since it generally does
not hold if depth and width tend to infinity simultaneously. In this paper, we
study the NTK of fully-connected ReLU networks with depth comparable to width.
We prove that the NTK properties depend significantly on the depth-to-width
ratio and the distribution of parameters at initialization. In fact, our
results indicate the importance of the three phases in the hyperparameter space
identified in Poole et al. (2016): ordered, chaotic and the edge of chaos
(EOC). We derive exact expressions for the NTK dispersion in the
infinite-depth-and-width limit in all three phases and conclude that the NTK
variability grows exponentially with depth at the EOC and in the chaotic phase
but not in the ordered phase. We also show that the NTK of deep networks may
stay constant during training only in the ordered phase and discuss how the
structure of the NTK matrix changes during training.

本文研究了深度与宽度相当的全连接 ReLU 网络的神经切向核（Neural Tangent Kernel）及其性质，发现其性质取决于深度与宽度之比以及初始状态下参数分布的情况。结果表明，在超参数空间中，有序、混沌和混沌边缘三个阶段很重要。在混沌和混沌边缘阶段，NTK 可变性随着深度呈指数增长，但在有序阶段则不会，此外还展示了深度神经网络的 NTK 只有在有序阶段中才能在训练过程中保持恒定，并探讨了 NTK 矩阵在训练过程中的结构变化。

深度和初始化对无限宽神经切向核的影响

Neural Tangent Kernel Beyond the Infinite-Width Limit: Effects of Depth and Initialization

After their successful debut in natural language processing, Transformer
architectures are now becoming the de-facto standard in many domains. An
obstacle for their deployment over new modalities is the architectural
configuration: the optimal depth-to-width ratio has been shown to dramatically
vary across data types (e.g., $10$x larger over images than over language). We
theoretically predict the existence of an embedding rank bottleneck that limits
the contribution of self-attention width to the Transformer expressivity. We
thus directly tie the input vocabulary size and rank to the optimal
depth-to-width ratio, since a small vocabulary size or rank dictates an added
advantage of depth over width. We empirically demonstrate the existence of this
bottleneck and its implications on the depth-to-width interplay of Transformer
architectures, linking the architecture variability across domains to the often
glossed-over usage of different vocabulary sizes or embedding ranks in
different domains. As an additional benefit, our rank bottlenecking framework
allows us to identify size redundancies of $25\%-50\%$ in leading NLP models
such as ALBERT and T5.

论文研究了 Transformer 的结构配置问题，并且提出了嵌入秩瓶颈的概念，通过实验验证了这种瓶颈的存在并显示了它在 Transformer 结构的深度与宽度之间的相互作用中的影响，同时提出了一种排查 ALBERT 和 T5 在 NLP 模型中冗余的方法。