The widespread use of neural networks across different scientific domains
often involves constraining them to satisfy certain symmetries, conservation
laws, or other domain knowledge. Such constraints are often imposed as soft
penalties during model training and effectively act as domain-specific
regularizers of the empirical risk loss. Physics-informed neural networks is an
example of this philosophy in which the outputs of deep neural networks are
constrained to approximately satisfy a given set of partial differential
equations. In this work we review recent advances in scientific machine
learning with a specific focus on the effectiveness of physics-informed neural
networks in predicting outcomes of physical systems and discovering hidden
physics from noisy data. We will also identify and analyze a fundamental mode
of failure of such approaches that is related to numerical stiffness leading to
unbalanced back-propagated gradients during model training. To address this
limitation we present a learning rate annealing algorithm that utilizes
gradient statistics during model training to balance the interplay between
different terms in composite loss functions. We also propose a novel neural
network architecture that is more resilient to such gradient pathologies. Taken
together, our developments provide new insights into the training of
constrained neural networks and consistently improve the predictive accuracy of
physics-informed neural networks by a factor of 50-100x across a range of
problems in computational physics. All code and data accompanying this
manuscript are publicly available at
https://github.com/PredictiveIntelligenceLab/GradientPathologiesPINNs.

本文回顾了科学机器学习的最新进展，特别关注物理启发式神经网络在预测物理系统结果和从噪声数据中发现隐藏的物理方面的有效性。我们提出了一个神经网络体系结构，该结构对梯度病理具有更强的鲁棒性，并提出了一个学习率退火算法，通过使用模型训练期间的梯度统计数据来平衡复合损失函数中不同项之间的相互作用。所有伴随本手稿的代码和数据都是公开可用的。