Neural collapse provides an elegant mathematical characterization of learned
last layer representations (a.k.a. features) and classifier weights in deep
classification models. Such results not only provide insights but also motivate
new techniques for improving practical deep models. However, most of the
existing empirical and theoretical studies in neural collapse focus on the case
that the number of classes is small relative to the dimension of the feature
space. This paper extends neural collapse to cases where the number of classes
are much larger than the dimension of feature space, which broadly occur for
language models, retrieval systems, and face recognition applications. We show
that the features and classifier exhibit a generalized neural collapse
phenomenon, where the minimum one-vs-rest margins is maximized.We provide
empirical study to verify the occurrence of generalized neural collapse in
practical deep neural networks. Moreover, we provide theoretical study to show
that the generalized neural collapse provably occurs under unconstrained
feature model with spherical constraint, under certain technical conditions on
feature dimension and number of classes.

该论文扩展了神经崩溃理论到类别数量远大于特征空间维度的情况，并提供了实证和理论研究以验证广义神经崩溃现象的存在和原理。

大规模多类别问题的广义神经塌陷

Generalized Neural Collapse for a Large Number of Classes

In this paper, we extends original Neural Collapse Phenomenon by proving
Generalized Neural Collapse hypothesis. We obtain Grassmannian Frame structure
from the optimization and generalization of classification. This structure
maximally separates features of every two classes on a sphere and does not
require a larger feature dimension than the number of classes. Out of curiosity
about the symmetry of Grassmannian Frame, we conduct experiments to explore if
models with different Grassmannian Frames have different performance. As a
result, we discover the Symmetric Generalization phenomenon. We provide a
theorem to explain Symmetric Generalization of permutation. However, the
question of why different directions of features can lead to such different
generalization is still open for future investigation.

本文提出了广义神经崩溃假说，并通过优化和分类的推广获得 Grassmann 框架结构。该结构在球面上最大化地分离每两类的特征，不需要比类别数量更大的特征维度。我们发现了对称泛化现象，并提供了一个用于解释置换的对称泛化定理。尽管不同特征方向导致不同泛化的原因仍需未来研究。

神经塌陷现象的研究：Grassmann 框架，对称性，泛化

A Study of Neural Collapse Phenomenon: Grassmannian Frame, Symmetry,  Generalization

The neural collapse (NC) phenomenon describes an underlying geometric
symmetry for deep neural networks, where both deeply learned features and
classifiers converge to a simplex equiangular tight frame. It has been shown
that both cross-entropy loss and mean square error can provably lead to NC. We
remove NC's key assumption on the feature dimension and the number of classes,
and then present a generalized neural collapse (GNC) hypothesis that
effectively subsumes the original NC. Inspired by how NC characterizes the
training target of neural networks, we decouple GNC into two objectives:
minimal intra-class variability and maximal inter-class separability. We then
use hyperspherical uniformity (which characterizes the degree of uniformity on
the unit hypersphere) as a unified framework to quantify these two objectives.
Finally, we propose a general objective -- hyperspherical uniformity gap (HUG),
which is defined by the difference between inter-class and intra-class
hyperspherical uniformity. HUG not only provably converges to GNC, but also
decouples GNC into two separate objectives. Unlike cross-entropy loss that
couples intra-class compactness and inter-class separability, HUG enjoys more
flexibility and serves as a good alternative loss function. Empirical results
show that HUG works well in terms of generalization and robustness.

该研究提出了广义神经坍塌假设（GNC），使用超球统一性作为衡量内部类别可变性和外部类别可分性的框架，提出一种泛化目标 HUG，通过测量 GNC 和 HUG 之间的差异来解耦 GNC，作为一种有效的交叉墒代价函数的替代函数。