Training neural networks sequentially in time to approximate solution fields
of time-dependent partial differential equations can be beneficial for
preserving causality and other physics properties; however, the
sequential-in-time training is numerically challenging because training errors
quickly accumulate and amplify over time. This work introduces Neural Galerkin
schemes that update randomized sparse subsets of network parameters at each
time step. The randomization avoids overfitting locally in time and so helps
prevent the error from accumulating quickly over the sequential-in-time
training, which is motivated by dropout that addresses a similar issue of
overfitting due to neuron co-adaptation. The sparsity of the update reduces the
computational costs of training without losing expressiveness because many of
the network parameters are redundant locally at each time step. In numerical
experiments with a wide range of evolution equations, the proposed scheme with
randomized sparse updates is up to two orders of magnitude more accurate at a
fixed computational budget and up to two orders of magnitude faster at a fixed
accuracy than schemes with dense updates.

通过在时序中顺序训练神经网络来逼近时变偏微分方程的解场，有助于保持因果性和其他物理特性；然而，由于训练误差随时间快速累积和放大，时序训练的数值挑战很大。本研究引入了 Neural Galerkin 方案，该方案在每个时间步骤更新网络参数的随机稀疏子集。随机化的操作避免了局部时间上的过拟合问题，因此有助于防止误差在时序训练中快速累积。该方案的更新稀疏性减少了训练的计算成本，而不会失去表达能力，因为在每个时间步骤中，很多网络参数是冗余的。通过与密集更新方案进行的大量数值实验表明，使用随机稀疏更新的建议方案在固定计算预算下精度高出两个数量级，而在固定精度下速度快两个数量级。