Neural Collapse refers to the remarkable structural properties characterizing the geometry of class embeddings and classifier weights, found by deep nets when trained beyond zero training error. However, this characterization only holds for balanced data. Here we thus ask whether it can be made invariant to class imbalances. Towards this end, we adopt the unconstrained-features model (UFM), a recent theoretical model for studying neural collapse, and introduce Simplex-Encoded-Labels Interpolation (SELI) as an invariant characterization of the neural collapse phenomenon. Specifically, we prove for the UFM with cross-entropy loss and vanishing regularization that, irrespective of class imbalances, the embeddings and classifiers always interpolate a simplex-encoded label matrix and that their individual geometries are determined by the SVD factors of this same label matrix. We then present extensive experiments on synthetic and real datasets that confirm convergence to the SELI geometry. However, we caution that convergence worsens with increasing imbalances. We theoretically support this finding by showing that unlike the balanced case, when minorities are present, ridge-regularization plays a critical role in tweaking the geometry. This defines new questions and motivates further investigations into the impact of class imbalances on the rates at which first-order methods converge to their asymptotically preferred solutions.

本文通过使用无限制特征模型和引入Simplex编码标签插值（SELI）作为神经坍塌现象的不变特征，证明了在使用交叉熵损失和消失正则化时，不受类别不平衡的影响，分类器总是插值出一个Simplex编码标签矩阵，而它们的几何形状取决于相同标签矩阵的SVD因子。同时，实验结果也表明收敛速度会随着类别不平衡的增加而变慢，并证明正则化对于修正分类器的几何形状也有着至关重要的作用。

失衡问题：重新审视神经陷入坍塌的几何特性