Neural collapse is a highly symmetric geometric pattern of neural networks
that emerges during the terminal phase of training, with profound implications
on the generalization performance and robustness of the trained networks. To
understand how the last-layer features and classifiers exhibit this recently
discovered implicit bias, in this paper, we introduce a surrogate model called
the unconstrained layer-peeled model (ULPM). We prove that gradient flow on
this model converges to critical points of a minimum-norm separation problem
exhibiting neural collapse in its global minimizer. Moreover, we show that the
ULPM with the cross-entropy loss has a benign global landscape for its loss
function, which allows us to prove that all the critical points are strict
saddle points except the global minimizers that exhibit the neural collapse
phenomenon. Empirically, we show that our results also hold during the training
of neural networks in real-world tasks when explicit regularization or weight
decay is not used.

本文研究神经网络的一种几何模式 —— 神经坍塌现象。我们提出了一个模型 —— 无约束层剥模型（ULPM），证明了该模型的梯度流收敛到一个最小范数分离问题的临界点，表现出全局最小化时神经坍塌现象。此外，我们还表明，采用交叉熵损失函数的 ULPM 具有良好的全局景观，所有临界点都是严格鞍点，除全局最小值外不发生神经坍塌现象。我们的实验证明，即使不使用显式正则化或权重衰减，我们的结果也适用于神经网络的实际任务训练。