Physics-informed neural networks (PINNs) are infamous for being hard to
train. Recently, second-order methods based on natural gradient and
Gauss-Newton methods have shown promising performance, improving the accuracy
achieved by first-order methods by several orders of magnitude. While
promising, the proposed methods only scale to networks with a few thousand
parameters due to the high computational cost to evaluate, store, and invert
the curvature matrix. We propose Kronecker-factored approximate curvature
(KFAC) for PINN losses that greatly reduces the computational cost and allows
scaling to much larger networks. Our approach goes beyond the established KFAC
for traditional deep learning problems as it captures contributions from a
PDE's differential operator that are crucial for optimization. To establish
KFAC for such losses, we use Taylor-mode automatic differentiation to describe
the differential operator's computation graph as a forward network with shared
weights. This allows us to apply KFAC thanks to a recently-developed general
formulation for networks with weight sharing. Empirically, we find that our
KFAC-based optimizers are competitive with expensive second-order methods on
small problems, scale more favorably to higher-dimensional neural networks and
PDEs, and consistently outperform first-order methods and LBFGS.

提出了 Krondocker 因式近似曲率法（KFAC）来降低计算成本，使得 PINN 模型能够扩展到更大的网络规模。此方法在小规模问题上与昂贵的二阶方法竞争，对于更高维度的神经网络和 PDEs 有更好的扩展能力，并持续优于一阶方法和 LBFGS。

物理知识神经网络的克罗内克因式化近似曲率

Kronecker-Factored Approximate Curvature for Physics-Informed Neural  Networks

Second-order methods for deep learning -- such as KFAC -- can be useful for
neural net training. However, they are often memory-inefficient and numerically
unstable for low-precision training since their preconditioning Kronecker
factors are dense, and require high-precision matrix inversion or
decomposition. Consequently, such methods are not widely used for training
large neural networks such as transformer-based models. We address these two
issues by (i) formulating an inverse-free update of KFAC and (ii) imposing
structures in each of the Kronecker factors, resulting in a method we term
structured inverse-free natural gradient descent (SINGD). On large modern
neural networks, we show that, in contrast to KFAC, SINGD is memory efficient
and numerically robust, and often outperforms AdamW even in half precision.
Hence, our work closes a gap between first-order and second-order methods in
modern low precision training for large neural nets.

用结构化的无逆自然梯度下降方法（SINGD）解决了 KFAC 方法在低精度训练中的内存低效和数值不稳定的问题，并在大型神经网络上表现出很好的性能，甚至在半精度上常常优于 AdamW。

结构化逆快速自然梯度：大型神经网络的内存高效和数值稳定 KFAC

Structured Inverse-Free Natural Gradient: Memory-Efficient &  Numerically-Stable KFAC for Large Neural Nets

Numerous researchers recently applied empirical spectral analysis to the
study of modern deep learning classifiers. We identify and discuss an important
formal class/cross-class structure and show how it lies at the origin of the
many visually striking features observed in deepnet spectra, some of which were
reported in recent articles, others are unveiled here for the first time. These
include spectral outliers, "spikes", and small but distinct continuous
distributions, "bumps", often seen beyond the edge of a "main bulk".
The significance of the cross-class structure is illustrated in three ways:
(i) we prove the ratio of outliers to bulk in the spectrum of the Fisher
information matrix is predictive of misclassification, in the context of
multinomial logistic regression; (ii) we demonstrate how, gradually with depth,
a network is able to separate class-distinctive information from class
variability, all while orthogonalizing the class-distinctive information; and
(iii) we propose a correction to KFAC, a well-known second-order optimization
algorithm for training deepnets.

研究应用实证频谱分析于现代深度学习分类器，展示其在深度网络频谱中的重要形式类 / 跨类结构及其特征，包括频谱异常值、突起和小但清晰的连续分布，介绍跨类结构的重要性，并证明多项式逻辑回归中，Fisher 信息矩阵频谱中异常值与主体审查的比例预测了误分类，在深度上，网络能够在将类别区分信息正交化的同时，逐渐分离类别特定信息，并提出了 KFAC 的校正。