A novel comparison is presented of the effect of optimiser choice on the
accuracy of physics-informed neural networks (PINNs). To give insight into why
some optimisers are better, a new approach is proposed that tracks the training
trajectory curvature and can be evaluated on the fly at a low computational
cost. The linear advection equation is studied for several advective
velocities, and we show that the optimiser choice substantially impacts PINNs
model performance and accuracy. Furthermore, using the curvature measure, we
found a negative correlation between the convergence error and the curvature in
the optimiser local reference frame. It is concluded that, in this case, larger
local curvature values result in better solutions. Consequently, optimisation
of PINNs is made more difficult as minima are in highly curved regions.

比较了不同优化器对物理知识驱动神经网络（PINNs）精度的影响，并提出了一种新的方法，跟踪训练轨迹曲率以解释为什么某些优化器效果更好。通过研究线性平流方程确定了优化器选择对 PINNs 模型性能和精度的影响，同时发现收敛误差与优化器本地参考系中的曲率存在负相关。总结认为，在此案例中，较大的本地曲率值会产生更好的解决方案，因此，在高曲率区域中出现极小值会使 PINNs 的优化变得更加困难。

物理约束神经网络中的优化探究

Probing optimisation in physics-informed neural networks

We propose Multi-Level Local SGD, a distributed gradient method for learning
a smooth, non-convex objective in a heterogeneous multi-level network. Our
network model consists of a set of disjoint sub-networks, with a single hub and
multiple worker nodes; further, worker nodes may have different operating
rates. The hubs exchange information with one another via a connected, but not
necessarily complete communication network. In our algorithm, sub-networks
execute a distributed SGD algorithm, using a hub-and-spoke paradigm, and the
hubs periodically average their models with neighboring hubs. We first provide
a unified mathematical framework that describes the Multi-Level Local SGD
algorithm. We then present a theoretical analysis of the algorithm; our
analysis shows the dependence of the convergence error on the worker node
heterogeneity, hub network topology, and the number of local, sub-network, and
global iterations. We back up our theoretical results via simulation-based
experiments using both convex and non-convex objectives.

我们提出了一种分布式梯度方法 ——Multi-Level Local SGD，用于在异构多级网络中学习一个平滑的、非凸的目标。

多层次本地 SGD 用于异构层次网络

Multi-Level Local SGD for Heterogeneous Hierarchical Networks

Distributed Stochastic Gradient Descent (SGD) when run in a synchronous
manner, suffers from delays in runtime as it waits for the slowest workers
(stragglers). Asynchronous methods can alleviate stragglers, but cause gradient
staleness that can adversely affect the convergence error. In this work, we
present a novel theoretical characterization of the speedup offered by
asynchronous methods by analyzing the trade-off between the error in the
trained model and the actual training runtime(wallclock time). The main novelty
in our work is that our runtime analysis considers random straggling delays,
which helps us design and compare distributed SGD algorithms that strike a
balance between straggling and staleness. We also provide a new error
convergence analysis of asynchronous SGD variants without bounded or
exponential delay assumptions. Finally, based on our theoretical
characterization of the error-runtime trade-off, we propose a method of
gradually varying synchronicity in distributed SGD and demonstrate its
performance on CIFAR10 dataset.

本研究分析同步和异步分布式随机梯度下降算法的误差和训练时间之间的权衡，考虑到随机拖延延迟，提出了逐渐变化同步性的方法，并在 CIFAR10 数据集上表现良好。