Deep neural networks (DNNs) exhibit a surprising structure in their final
layer known as neural collapse (NC), and a growing body of works has currently
investigated the propagation of neural collapse to earlier layers of DNNs -- a
phenomenon called deep neural collapse (DNC). However, existing theoretical
results are restricted to special cases: linear models, only two layers or
binary classification. In contrast, we focus on non-linear models of arbitrary
depth in multi-class classification and reveal a surprising qualitative shift.
As soon as we go beyond two layers or two classes, DNC stops being optimal for
the deep unconstrained features model (DUFM) -- the standard theoretical
framework for the analysis of collapse. The main culprit is a low-rank bias of
multi-layer regularization schemes: this bias leads to optimal solutions of
even lower rank than the neural collapse. We support our theoretical findings
with experiments on both DUFM and real data, which show the emergence of the
low-rank structure in the solution found by gradient descent.

深度神经网络中的神经塌缩现象对于非线性模型和多分类问题呈现低秩结构的突变是其优化的主要原因。

神经崩溃与低秩偏差：深度神经网络崩溃是否真正最优？

Neural Collapse versus Low-rank Bias: Is Deep Neural Collapse Really  Optimal?

Modern deep neural networks have achieved high performance across various
tasks. Recently, researchers have noted occurrences of low-dimensional
structure in the weights, Hessian's, gradients, and feature vectors of these
networks, spanning different datasets and architectures when trained to
convergence. In this analysis, we theoretically demonstrate these observations
arising, and show how they can be unified within a generalized unconstrained
feature model that can be considered analytically. Specifically, we consider a
previously described structure called Neural Collapse, and its multi-layer
counterpart, Deep Neural Collapse, which emerges when the network approaches
global optima. This phenomenon explains the other observed low-dimensional
behaviours on a layer-wise level, such as the bulk and outlier structure seen
in Hessian spectra, and the alignment of gradient descent with the outlier
eigenspace of the Hessian. Empirical results in both the deep linear
unconstrained feature model and its non-linear equivalent support these
predicted observations.

现代深度神经网络在各种任务中取得了高性能，研究人员最近注意到这些网络的权重、Hessian 矩阵、梯度和特征向量中存在低维结构，这种低维结构在不同的数据集和架构上进行训练时都能观察到。在本文中，我们在理论上证明了这些观察结果的发生，并展示了如何在一个可以被分析地考虑的广义非约束特征模型中统一这些观察结果。特别地，我们考虑了一种先前描述的结构，称为神经坍缩，以及它的多层对应物，即深度神经坍缩，在网络接近全局最优解时出现。这种现象解释了观察到的其他低维行为，比如在 Hessian 谱中观察到的批量和离群结构，以及梯度下降与 Hessian 的离群特征空间的对齐。深度线性非约束特征模型及其非线性等效模型的经验结果支持了这些预测的观察结果。

深度线性无约束特征模型在深度学习中统一低维观测

Unifying Low Dimensional Observations in Deep Learning Through the Deep  Linear Unconstrained Feature Model

Deep Neural Collapse (DNC) refers to the surprisingly rigid structure of the
data representations in the final layers of Deep Neural Networks (DNNs). Though
the phenomenon has been measured in a wide variety of settings, its emergence
is only partially understood. In this work, we provide substantial evidence
that DNC formation occurs primarily through deep feature learning with the
average gradient outer product (AGOP). This takes a step further compared to
efforts that explain neural collapse via feature-agnostic approaches, such as
the unconstrained features model. We proceed by providing evidence that the
right singular vectors and values of the weights are responsible for the
majority of within-class variability collapse in DNNs. As shown in recent work,
this singular structure is highly correlated with that of the AGOP. We then
establish experimentally and theoretically that AGOP induces neural collapse in
a randomly initialized neural network. In particular, we demonstrate that Deep
Recursive Feature Machines, a method originally introduced as an abstraction
for AGOP feature learning in convolutional neural networks, exhibits DNC.

Deep Neural Collapse（DNC）是深度神经网络（DNNs）中最终层数据表示的令人惊讶的刚性结构。本研究提供了实质性的证据，表明 DNC 主要通过平均梯度外积（AGOP）进行深度特征学习形成。我们证明了权重的右奇异向量和奇异值是 DNNs 中类内变异性大部分倒塌的原因之一，并与 AGOP 的奇异结构高度相关。此外，我们实验和理论上证明了 AGOP 在随机初始化的神经网络中引起神经崩溃。由此证明，Deep Recursive Feature Machines 是 AGOP 在卷积神经网络中进行特征学习的抽象方法，并且表现出 DNC 现象。