An intriguing phenomenon observed during training neural networks is the
spectral bias, which states that neural networks are biased towards learning
less complex functions. The priority of learning functions with low complexity
might be at the core of explaining generalization ability of neural network,
and certain efforts have been made to provide theoretical explanation for
spectral bias. However, there is still no satisfying theoretical result
justifying the underlying mechanism of spectral bias. In this paper, we give a
comprehensive and rigorous explanation for spectral bias and relate it with the
neural tangent kernel function proposed in recent work. We prove that the
training process of neural networks can be decomposed along different
directions defined by the eigenfunctions of the neural tangent kernel, where
each direction has its own convergence rate and the rate is determined by the
corresponding eigenvalue. We then provide a case study when the input data is
uniformly distributed over the unit sphere, and show that lower degree
spherical harmonics are easier to be learned by over-parameterized neural
networks. Finally, we provide numerical experiments to demonstrate the
correctness of our theory. Our experimental results also show that our theory
can tolerate certain model misspecification in terms of the input data
distribution.

论文通过与神经切向核函数的关联性进行解释，给出了关于神经网络中谱偏置的全面而严格的解释，并证明了神经网络的训练过程可以沿神经切向核的不同方向分解，每个方向都有自己的收敛率，而该收敛率由相应的特征值确定。