Using neural networks to solve partial differential equations (PDEs) is
gaining popularity as an alternative approach in the scientific computing
community. Neural networks can integrate different types of information into
the loss function. These include observation data, governing equations, and
variational forms, etc. These loss functions can be broadly categorized into
two types: observation data loss directly constrains and measures the model
output, while other loss functions indirectly model the performance of the
network, which can be classified as model loss. However, this alternative
approach lacks a thorough understanding of its underlying mechanisms, including
theoretical foundations and rigorous characterization of various phenomena.
This work focuses on investigating how different loss functions impact the
training of neural networks for solving PDEs. We discover a stable loss-jump
phenomenon: when switching the loss function from the data loss to the model
loss, which includes different orders of derivative information, the neural
network solution significantly deviates from the exact solution immediately.
Further experiments reveal that this phenomenon arises from the different
frequency preferences of neural networks under different loss functions. We
theoretically analyze the frequency preference of neural networks under model
loss. This loss-jump phenomenon provides a valuable perspective for examining
the underlying mechanisms of neural networks in solving PDEs.

通过研究不同损失函数对神经网络解决偏微分方程的训练的影响，我们发现从数据损失到模型损失切换时会出现稳定的损失跃变现象， 进一步实验证明这一现象源于神经网络在不同损失函数下的频率偏好，这为研究神经网络在解决偏微分方程时的内在机制提供了有价值的视角。