This study delves into the intricate dynamics of trained deep neural networks
and their relationships with network parameters. Trained networks predominantly
continue training in a single direction, known as the drift mode. This drift
mode can be explained by the quadratic potential model of the loss function,
suggesting a slow exponential decay towards the potential minima. We unveil a
correlation between Hessian eigenvectors and network weights. This
relationship, hinging on the magnitude of eigenvalues, allows us to discern
parameter directions within the network. Notably, the significance of these
directions relies on two defining attributes: the curvature of their potential
wells (indicated by the magnitude of Hessian eigenvalues) and their alignment
with the weight vectors. Our exploration extends to the decomposition of weight
matrices through singular value decomposition. This approach proves practical
in identifying critical directions within the Hessian, considering both their
magnitude and curvature. Furthermore, our examination showcases the
applicability of principal component analysis in approximating the Hessian,
with update parameters emerging as a superior choice over weights for this
purpose. Remarkably, our findings unveil a similarity between the largest
Hessian eigenvalues of individual layers and the entire network. Notably,
higher eigenvalues are concentrated more in deeper layers. Leveraging these
insights, we venture into addressing catastrophic forgetting, a challenge of
neural networks when learning new tasks while retaining knowledge from previous
ones. By applying our discoveries, we formulate an effective strategy to
mitigate catastrophic forgetting, offering a possible solution that can be
applied to networks of varying scales, including larger architectures.

该研究探讨了训练深度神经网络及其与网络参数之间的复杂动力学关系。通过研究我们发现，训练网络往往沿着单一方向进行训练，被称为漂移模式。通过损失函数的二次势模型，我们解释了这种漂移模式，并提出其向潜在值的指数级缓慢衰减。我们揭示了 Hessian 特征向量与网络权重之间的相关性，该关系取决于特征值的大小，使我们能够识别网络内的参数方向。此外，通过奇异值分解，我们对权重矩阵进行了分解，以实用的方式识别 Hessian 内的关键方向，同时考虑其大小和曲率。此外，我们还发现了各层最大的 Hessian 特征值与整个网络之间的相似性，特别是更大的特征值更集中在深层。最后，基于我们的发现，我们探索了解决神经网络在学习新任务时遗忘之前任务知识的挑战，通过应用我们的发现，我们提出了一种有效的策略来缓解这种遗忘，这个策略可以适用于不同规模的网络，包括更大的架构。

神经网络权重矩阵的 Hessian 特征向量和主成分分析

Hessian Eigenvectors and Principal Component Analysis of Neural Network  Weight Matrices

We show that in a variety of large-scale deep learning scenarios the gradient
dynamically converges to a very small subspace after a short period of
training. The subspace is spanned by a few top eigenvectors of the Hessian
(equal to the number of classes in the dataset), and is mostly preserved over
long periods of training. A simple argument then suggests that gradient descent
may happen mostly in this subspace. We give an example of this effect in a
solvable model of classification, and we comment on possible implications for
optimization and learning.

本文研究表明，在大规模深度学习场景中，梯度在短时间训练后动态地收敛到一个非常小的子空间。该子空间由海森矩阵的前几个特征向量构成，并在长时间训练中大部分保持不变。我们给出了一个分类问题可解模型的例子，并对此效应对优化和学习的潜在影响进行了评论。