Deep Neural Collapse (DNC) refers to the surprisingly rigid structure of the
data representations in the final layers of Deep Neural Networks (DNNs). Though
the phenomenon has been measured in a wide variety of settings, its emergence
is only partially understood. In this work, we provide substantial evidence
that DNC formation occurs primarily through deep feature learning with the
average gradient outer product (AGOP). This takes a step further compared to
efforts that explain neural collapse via feature-agnostic approaches, such as
the unconstrained features model. We proceed by providing evidence that the
right singular vectors and values of the weights are responsible for the
majority of within-class variability collapse in DNNs. As shown in recent work,
this singular structure is highly correlated with that of the AGOP. We then
establish experimentally and theoretically that AGOP induces neural collapse in
a randomly initialized neural network. In particular, we demonstrate that Deep
Recursive Feature Machines, a method originally introduced as an abstraction
for AGOP feature learning in convolutional neural networks, exhibits DNC.

Deep Neural Collapse（DNC）是深度神经网络（DNNs）中最终层数据表示的令人惊讶的刚性结构。本研究提供了实质性的证据，表明 DNC 主要通过平均梯度外积（AGOP）进行深度特征学习形成。我们证明了权重的右奇异向量和奇异值是 DNNs 中类内变异性大部分倒塌的原因之一，并与 AGOP 的奇异结构高度相关。此外，我们实验和理论上证明了 AGOP 在随机初始化的神经网络中引起神经崩溃。由此证明，Deep Recursive Feature Machines 是 AGOP 在卷积神经网络中进行特征学习的抽象方法，并且表现出 DNC 现象。

平均梯度外积：作为深度神经崩溃机制

Average gradient outer product as a mechanism for deep neural collapse

We present a large-scale collection of diverse natural language inference
(NLI) datasets that help provide insight into how well a sentence
representation captures distinct types of reasoning. The collection results
from recasting 13 existing datasets from 7 semantic phenomena into a common NLI
structure, resulting in over half a million labeled context-hypothesis pairs in
total. We refer to our collection as the DNC: Diverse Natural Language
Inference Collection. The DNC is available online at this https URL,
and will grow over time as additional resources are recast and added from novel
sources.

该研究从 13 个不同领域的语料库中抽取出超过 50 万个标记好的内容假设对，并将它们重新组织成一个名为 DNC 的大规模多样化自然语言推理数据集，以探究语句表示在不同类型推理问题中的表现。