Recent works have shown that traditional Neural Network (NN) architectures
display a marked frequency bias in the learning process. Namely, the NN first
learns the low-frequency features before learning the high-frequency ones. In
this study, we rigorously develop a partial differential equation (PDE) that
unravels the frequency dynamics of the error for a 2-layer NN in the Neural
Tangent Kernel regime. Furthermore, using this insight, we explicitly
demonstrate how an appropriate choice of distributions for the initialization
weights can eliminate or control the frequency bias. We focus our study on the
Fourier Features model, an NN where the first layer has sine and cosine
activation functions, with frequencies sampled from a prescribed distribution.
In this setup, we experimentally validate our theoretical results and compare
the NN dynamics to the solution of the PDE using the finite element method.
Finally, we empirically show that the same principle extends to multi-layer
NNs.

传统神经网络在学习过程中存在频率偏差，本研究通过偏微分方程研究了神经网络中错误频率的动力学，进一步证明了通过适当选择初始化权重的分布可以消除或控制频率偏差，并实验证实了该原理也适用于多层神经网络。