Recent works have partly attributed the generalization ability of
over-parameterized neural networks to frequency bias -- networks trained with
gradient descent on data drawn from a uniform distribution find a low frequency
fit before high frequency ones. As realistic training sets are not drawn from a
uniform distribution, we here use the Neural Tangent Kernel (NTK) model to
explore the effect of variable density on training dynamics. Our results, which
combine analytic and empirical observations, show that when learning a pure
harmonic function of frequency $\kappa$, convergence at a point $\x \in
\Sphere^{d-1}$ occurs in time $O(\kappa^d/p(\x))$ where $p(\x)$ denotes the
local density at $\x$. Specifically, for data in $\Sphere^1$ we analytically
derive the eigenfunctions of the kernel associated with the NTK for two-layer
networks. We further prove convergence results for deep, fully connected
networks with respect to the spectral decomposition of the NTK. Our empirical
study highlights similarities and differences between deep and shallow networks
in this model.

本文使用神经切向核（NTK）模型研究变量密度对训练动态的影响，结果表明在学习频率为 k 的纯谐波函数时，对于数据在 S^1 上，收敛点 x 的时间复杂度为 O (k^d/p (x))，其中 p (x) 表示 x 处的局部密度。