Neural collapse (NC) refers to the surprising structure of the last layer of
deep neural networks in the terminal phase of gradient descent training.
Recently, an increasing amount of experimental evidence has pointed to the
propagation of NC to earlier layers of neural networks. However, while the NC
in the last layer is well studied theoretically, much less is known about its
multi-layered counterpart - deep neural collapse (DNC). In particular, existing
work focuses either on linear layers or only on the last two layers at the
price of an extra assumption. Our paper fills this gap by generalizing the
established analytical framework for NC - the unconstrained features model - to
multiple non-linear layers. Our key technical contribution is to show that, in
a deep unconstrained features model, the unique global optimum for binary
classification exhibits all the properties typical of DNC. This explains the
existing experimental evidence of DNC. We also empirically show that (i) by
optimizing deep unconstrained features models via gradient descent, the
resulting solution agrees well with our theory, and (ii) trained networks
recover the unconstrained features suitable for the occurrence of DNC, thus
supporting the validity of this modeling principle.

本文将已有的理论框架扩展到多个非线性层，通过理论证明和实验验证，说明神经网络的最后一个层的可折叠性会向更早的层传播，同时解释了已有的实验结果。

深度神经网络崩塌对于深度非约束特征模型具有可证明的最优性

Deep Neural Collapse Is Provably Optimal for the Deep Unconstrained  Features Model

The modern strategy for training deep neural networks for classification
tasks includes optimizing the network's weights even after the training error
vanishes to further push the training loss toward zero. Recently, a phenomenon
termed "neural collapse" (NC) has been empirically observed in this training
procedure. Specifically, it has been shown that the learned features (the
output of the penultimate layer) of within-class samples converge to their
mean, and the means of different classes exhibit a certain tight frame
structure, which is also aligned with the last layer's weights. Recent papers
have shown that minimizers with this structure emerge when optimizing a
simplified "unconstrained features model" (UFM) with a regularized
cross-entropy loss. In this paper, we further analyze and extend the UFM.
First, we study the UFM for the regularized MSE loss, and show that the
minimizers' features can have a more delicate structure than in the
cross-entropy case. This affects also the structure of the weights. Then, we
extend the UFM by adding another layer of weights as well as ReLU nonlinearity
to the model and generalize our previous results. Finally, we empirically
demonstrate the usefulness of our nonlinear extended UFM in modeling the NC
phenomenon that occurs with practical networks.

本研究旨在进一步分析和扩展无约束特征模型（UFM）。我们发现，通过将另一层权重以及 ReLU 非线性加入模型，可以更好地描述神经崩溃现象，同时得出了一些实用的结论。

探索深度神经网络崩溃的扩展无约束特征模型

Extended Unconstrained Features Model for Exploring Deep Neural Collapse

Neural collapse is an emergent phenomenon in deep learning that was recently
discovered by Papyan, Han and Donoho. We propose a simple "unconstrained
features model" in which neural collapse also emerges empirically. By studying
this model, we provide some explanation for the emergence of neural collapse in
terms of the landscape of empirical risk.

本文介绍了一种简单的 “不受限特征模型”，该模型中神经网络崩溃现象得以实证，通过研究该模型，我们提供了关于神经网络崩溃产生原因的一些解释，从经验风险的角度进行解释。