The current paradigm of training deep neural networks for classification
tasks includes minimizing the empirical risk that pushes the training loss
value towards zero, even after the training error has been vanished. In this
terminal phase of training, it has been observed that the last-layer features
collapse to their class-means and these class-means converge to the vertices of
a simplex Equiangular Tight Frame (ETF). This phenomenon is termed as Neural
Collapse (NC). To theoretically understand this phenomenon, recent works employ
a simplified unconstrained feature model to prove that NC emerges at the global
solutions of the training problem. However, when the training dataset is
class-imbalanced, some NC properties will no longer be true. For example, the
class-means geometry will skew away from the simplex ETF when the loss
converges. In this paper, we generalize NC to imbalanced regime for
cross-entropy loss under the unconstrained ReLU feature model. We prove that,
while the within-class features collapse property still holds in this setting,
the class-means will converge to a structure consisting of orthogonal vectors
with different lengths. Furthermore, we find that the classifier weights are
aligned to the scaled and centered class-means with scaling factors depend on
the number of training samples of each class, which generalizes NC in the
class-balanced setting. We empirically prove our results through experiments on
practical architectures and dataset.

在该研究中，我们将神经坍缩现象推广到不平衡类别的交叉熵损失下，证明了虽然类内特征会坍缩，但类均值将收敛到长度不同的正交向量结构，并发现分类器权重与类均值的缩放和居中相关，这扩展了在平衡类别设置下的神经坍缩现象。通过在实际架构和数据集上进行实验证明了我们的结果。