Unrolling training trajectories over time strongly influences the inference
accuracy of neural network-augmented physics simulators. We analyze these
effects by studying three variants of training neural networks on discrete
ground truth trajectories. In addition to commonly used one-step setups and
fully differentiable unrolling, we include a third, less widely used variant:
unrolling without temporal gradients. Comparing networks trained with these
three modalities makes it possible to disentangle the two dominant effects of
unrolling, training distribution shift and long-term gradients. We present a
detailed study across physical systems, network sizes, network architectures,
training setups, and test scenarios. It provides an empirical basis for our
main findings: A non-differentiable but unrolled training setup supported by a
numerical solver can yield 4.5-fold improvements over a fully differentiable
prediction setup that does not utilize this solver. We also quantify a
difference in the accuracy of models trained in a fully differentiable setup
compared to their non-differentiable counterparts. While differentiable setups
perform best, the accuracy of unrolling without temporal gradients comes
comparatively close. Furthermore, we empirically show that these behaviors are
invariant to changes in the underlying physical system, the network
architecture and size, and the numerical scheme. These results motivate
integrating non-differentiable numerical simulators into training setups even
if full differentiability is unavailable. We also observe that the convergence
rate of common neural architectures is low compared to numerical algorithms.
This encourages the use of hybrid approaches combining neural and numerical
algorithms to utilize the benefits of both.

通过研究神经网络在离散地面真实轨迹上进行训练的三种变体，我们分析了时间上展开训练轨迹对物理模拟器推断准确性的影响，发现非可微分但展开的训练设置结合数值求解器能够比不利用该求解器的全可微分预测设置提高 4.5 倍准确性，而不带时间梯度的展开方法的准确性也相对接近可全可微分设置。此外，我们还观察到常见神经网络架构的收敛速率较低，鼓励使用融合神经和数值算法的混合方法来充分利用两者的优势。

时间展开对神经物理推理模型的支持

How Temporal Unrolling Supports Neural Physics Simulators

Recent works in deep learning have shown that integrating differentiable
physics simulators into the training process can greatly improve the quality of
results. Although this combination represents a more complex optimization task
than supervised neural network training, the same gradient-based optimizers are
typically employed to minimize the loss function. However, the integrated
physics solvers have a profound effect on the gradient flow as manipulating
scales in magnitude and direction is an inherent property of many physical
processes. Consequently, the gradient flow is often highly unbalanced and
creates an environment in which existing gradient-based optimizers perform
poorly. In this work, we analyze the characteristics of both physical and
neural network optimizations to derive a new method that does not suffer from
this phenomenon. Our method is based on a half-inversion of the Jacobian and
combines principles of both classical network and physics optimizers to solve
the combined optimization task. Compared to state-of-the-art neural network
optimizers, our method converges more quickly and yields better solutions,
which we demonstrate on three complex learning problems involving nonlinear
oscillators, the Schroedinger equation and the Poisson problem.

本研究分析了物理和神经网络的优化特征，并结合了经典网络和物理优化器的原则，基于 Jacobian 的半反演提出一种新方法，比目前最先进的神经网络优化器更快地收敛并产生更好的解决方案，此方法在非线性振荡器，Schroedinger 方程和泊松问题的三个复杂学习问题上得到了验证。