We derive analytical expressions for the generalization performance of kernel
regression as a function of the number of training samples using theoretical
methods from Gaussian processes and statistical physics. Our expressions apply
to wide neural networks due to an equivalence between training them and kernel
regression with the Neural Tangent Kernel (NTK). By computing the decomposition
of the total generalization error due to different spectral components of the
kernel, we identify a new spectral principle: as the size of the training set
grows, kernel machines and neural networks fit successively higher spectral
modes of the target function. When data are sampled from a uniform distribution
on a high-dimensional hypersphere, dot product kernels, including NTK, exhibit
learning stages where different frequency modes of the target function are
learned. We verify our theory with simulations on synthetic data and MNIST
dataset.

通过高斯过程和统计物理学的理论方法，我们得到了内核回归广义性能的分析表达式，这些表达式是关于训练样本数量的函数。我们的结果适用于具有广泛神经网络的情况，这是由于训练它们和使用神经切向核 (NTK) 的核回归之间的等效性。通过计算核的不同谱成分对总体泛化误差的分解，我们确定了一个新的谱原理：随着训练集大小的增长，核机和神经网络逐渐适应目标功能的更高频谱模式。当数据从高维超球面上的均匀分布中采样时，点积核，包括 NTK，显示出学习阶段，其中学习不同频率模式的目标函数。通过对合成数据和 MNIST 数据集的模拟，我们验证了我们的理论。