We consider a gradient-based optimization method applied to a function
$\mathcal{L}$ of a vector of variables $\boldsymbol{\theta}$, in the case where
$\boldsymbol{\theta}$ is represented as a tuple of tensors $(\mathbf{T}_1,
\cdots, \mathbf{T}_S)$. This framework encompasses many common use-cases, such
as training neural networks by gradient descent. First, we propose a
computationally inexpensive technique providing higher-order information on
$\mathcal{L}$, especially about the interactions between the tensors
$\mathbf{T}_s$, based on automatic differentiation and computational tricks.
Second, we use this technique at order 2 to build a second-order optimization
method which is suitable, among other things, for training deep neural networks
of various architectures. This second-order method leverages the partition
structure of $\boldsymbol{\theta}$ into tensors $(\mathbf{T}_1, \cdots,
\mathbf{T}_S)$, in such a way that it requires neither the computation of the
Hessian of $\mathcal{L}$ according to $\boldsymbol{\theta}$, nor any
approximation of it. The key part consists in computing a smaller matrix
interpretable as a "Hessian according to the partition", which can be computed
exactly and efficiently. In contrast to many existing practical second-order
methods used in neural networks, which perform a diagonal or block-diagonal
approximation of the Hessian or its inverse, the method we propose does not
neglect interactions between layers. Finally, we can tune the coarseness of the
partition to recover well-known optimization methods: the coarsest case
corresponds to Cauchy's steepest descent method, the finest case corresponds to
the usual Newton's method.

本研究论文介绍了一种基于梯度的优化方法，并提出了一个计算上廉价的技术，用于获得有关张量之间交互关系的二阶信息。使用这种技术，构建了适用于各种深度神经网络结构的二阶优化方法，避免了计算 Hessian 矩阵和其近似的复杂性，并改善了现有的对角线或块对角线近似方法。

通过高阶导数总结将牛顿法应用于神经网络

Adapting Newton's Method to Neural Networks through a Summary of  Higher-Order Derivatives

We present PYHESSIAN, a new scalable framework that enables fast computation
of Hessian (i.e., second-order derivative) information for deep neural
networks. PYHESSIAN enables fast computations of the top Hessian eigenvalues,
the Hessian trace, and the full Hessian eigenvalue/spectral density, and it
supports distributed-memory execution on cloud/supercomputer systems and is
available as open source. This general framework can be used to analyze neural
network models, including the topology of the loss landscape (i.e., curvature
information) to gain insight into the behavior of different models/optimizers.
To illustrate this, we analyze the effect of residual connections and Batch
Normalization layers on the trainability of neural networks. One recent claim,
based on simpler first-order analysis, is that residual connections and Batch
Normalization make the loss landscape smoother, thus making it easier for
Stochastic Gradient Descent to converge to a good solution. Our extensive
analysis shows new finer-scale insights, demonstrating that, while conventional
wisdom is sometimes validated, in other cases it is simply incorrect. In
particular, we find that Batch Normalization does not necessarily make the loss
landscape smoother, especially for shallower networks.

我们提出了 PYHESSIAN 框架，它可以快速计算深度神经网络的 Hessian 信息，支持分布式计算，并且可以用于分析神经网络模型，特别是损失函数曲率（即损失函数的拓扑），以便更好的理解不同模型和优化器的行为表现。通过对残差连接和 BN 层的分析，我们发现传统的方法不一定正确，BN 层不一定会使得损失函数曲率更加平滑，特别是在较浅的神经网络中。