We characterize and remedy a failure mode that may arise from multi-scale
dynamics with scale imbalances during training of deep neural networks, such as
Physics Informed Neural Networks (PINNs). PINNs are popular machine-learning
templates that allow for seamless integration of physical equation models with
data. Their training amounts to solving an optimization problem over a weighted
sum of data-fidelity and equation-fidelity objectives. Conflicts between
objectives can arise from scale imbalances, heteroscedasticity in the data,
stiffness of the physical equation, or from catastrophic interference during
sequential training. We explain the training pathology arising from this and
propose a simple yet effective inverse-Dirichlet weighting strategy to
alleviate the issue. We compare with Sobolev training of neural networks,
providing the baseline of analytically $\boldsymbol{\epsilon}$-optimal
training. We demonstrate the effectiveness of inverse-Dirichlet weighting in
various applications, including a multi-scale model of active turbulence, where
we show orders of magnitude improvement in accuracy and convergence over
conventional PINN training. For inverse modeling using sequential training, we
find that inverse-Dirichlet weighting protects a PINN against catastrophic
forgetting.

本文研究并解决了深度神经网络（如物理信息神经网络）训练过程中可能出现的多尺度动态和尺度失衡引发的故障模式，通过提出一种简单而有效的逆狄利克雷加权策略来缓解问题，并在各种应用中进行了验证和比较。