Deep Neural Collapse (DNC) refers to the surprisingly rigid structure of the
data representations in the final layers of Deep Neural Networks (DNNs). Though
the phenomenon has been measured in a wide variety of settings, its emergence
is only partially understood. In this work, we provide substantial evidence
that DNC formation occurs primarily through deep feature learning with the
average gradient outer product (AGOP). This takes a step further compared to
efforts that explain neural collapse via feature-agnostic approaches, such as
the unconstrained features model. We proceed by providing evidence that the
right singular vectors and values of the weights are responsible for the
majority of within-class variability collapse in DNNs. As shown in recent work,
this singular structure is highly correlated with that of the AGOP. We then
establish experimentally and theoretically that AGOP induces neural collapse in
a randomly initialized neural network. In particular, we demonstrate that Deep
Recursive Feature Machines, a method originally introduced as an abstraction
for AGOP feature learning in convolutional neural networks, exhibits DNC.

Deep Neural Collapse（DNC）是深度神经网络（DNNs）中最终层数据表示的令人惊讶的刚性结构。本研究提供了实质性的证据，表明 DNC 主要通过平均梯度外积（AGOP）进行深度特征学习形成。我们证明了权重的右奇异向量和奇异值是 DNNs 中类内变异性大部分倒塌的原因之一，并与 AGOP 的奇异结构高度相关。此外，我们实验和理论上证明了 AGOP 在随机初始化的神经网络中引起神经崩溃。由此证明，Deep Recursive Feature Machines 是 AGOP 在卷积神经网络中进行特征学习的抽象方法，并且表现出 DNC 现象。