The modern strategy for training deep neural networks for classification
tasks includes optimizing the network's weights even after the training error
vanishes to further push the training loss toward zero. Recently, a phenomenon
termed "neural collapse" (NC) has been empirically observed in this training
procedure. Specifically, it has been shown that the learned features (the
output of the penultimate layer) of within-class samples converge to their
mean, and the means of different classes exhibit a certain tight frame
structure, which is also aligned with the last layer's weights. Recent papers
have shown that minimizers with this structure emerge when optimizing a
simplified "unconstrained features model" (UFM) with a regularized
cross-entropy loss. In this paper, we further analyze and extend the UFM.
First, we study the UFM for the regularized MSE loss, and show that the
minimizers' features can have a more delicate structure than in the
cross-entropy case. This affects also the structure of the weights. Then, we
extend the UFM by adding another layer of weights as well as ReLU nonlinearity
to the model and generalize our previous results. Finally, we empirically
demonstrate the usefulness of our nonlinear extended UFM in modeling the NC
phenomenon that occurs with practical networks.

本研究旨在进一步分析和扩展无约束特征模型（UFM）。我们发现，通过将另一层权重以及 ReLU 非线性加入模型，可以更好地描述神经崩溃现象，同时得出了一些实用的结论。

探索深度神经网络崩溃的扩展无约束特征模型

Extended Unconstrained Features Model for Exploring Deep Neural Collapse

Understanding theoretical properties of deep and locally connected nonlinear
network, such as deep convolutional neural network (DCNN), is still a hard
problem despite its empirical success. In this paper, we propose a novel
theoretical framework for such networks with ReLU nonlinearity. The framework
explicitly formulates data distribution, favors disentangled representations
and is compatible with common regularization techniques such as Batch Norm. The
framework is built upon teacher-student setting, by expanding the student
forward/backward propagation onto the teacher's computational graph. The
resulting model does not impose unrealistic assumptions (e.g., Gaussian inputs,
independence of activation, etc). Our framework could help facilitate
theoretical analysis of many practical issues, e.g. overfitting,
generalization, disentangled representations in deep networks.

在深度卷积神经网络中，通过提出一种以带有 ReLU 非线性激活的网络为基础的新型理论框架，该框架通过在教师 - 学生设置中扩展学生的向前 / 向后传播，明确了数据分布，强调了分解表示，并且兼容常见的规则化技术，不会强加不现实的假设，这种框架有助于促进许多实用问题（如过拟合，概括，深度网络中的分解表示）的理论分析。