The ``Neural Tangent Kernel'' (NTK) (Jacot et al 2018), and its empirical
variants have been proposed as a proxy to capture certain behaviors of real
neural networks. In this work, we study NTKs through the lens of scaling laws,
and demonstrate that they fall short of explaining important aspects of neural
network generalization. In particular, we demonstrate realistic settings where
finite-width neural networks have significantly better data scaling exponents
as compared to their corresponding empirical and infinite NTKs at
initialization. This reveals a more fundamental difference between the real
networks and NTKs, beyond just a few percentage points of test accuracy.
Further, we show that even if the empirical NTK is allowed to be pre-trained on
a constant number of samples, the kernel scaling does not catch up to the
neural network scaling. Finally, we show that the empirical NTK continues to
evolve throughout most of the training, in contrast with prior work which
suggests that it stabilizes after a few epochs of training. Altogether, our
work establishes concrete limitations of the NTK approach in understanding
generalization of real networks on natural datasets.

本文通过缩放定律的角度研究神经切向核 (NTK) 及其经验性变量，发现它们无法完全解释神经网络泛化的重要方面。通过实际设置，我们展示了有限宽度神经网络相对于其对应的经验和无穷 NTK 起始时具有显着更好的数据缩放指数，并证明了 NTK 方法在理解自然数据集上真实网络泛化的局限性。