The neural collapse (NC) phenomenon describes an underlying geometric
symmetry for deep neural networks, where both deeply learned features and
classifiers converge to a simplex equiangular tight frame. It has been shown
that both cross-entropy loss and mean square error can provably lead to NC. We
remove NC's key assumption on the feature dimension and the number of classes,
and then present a generalized neural collapse (GNC) hypothesis that
effectively subsumes the original NC. Inspired by how NC characterizes the
training target of neural networks, we decouple GNC into two objectives:
minimal intra-class variability and maximal inter-class separability. We then
use hyperspherical uniformity (which characterizes the degree of uniformity on
the unit hypersphere) as a unified framework to quantify these two objectives.
Finally, we propose a general objective -- hyperspherical uniformity gap (HUG),
which is defined by the difference between inter-class and intra-class
hyperspherical uniformity. HUG not only provably converges to GNC, but also
decouples GNC into two separate objectives. Unlike cross-entropy loss that
couples intra-class compactness and inter-class separability, HUG enjoys more
flexibility and serves as a good alternative loss function. Empirical results
show that HUG works well in terms of generalization and robustness.

该研究提出了广义神经坍塌假设（GNC），使用超球统一性作为衡量内部类别可变性和外部类别可分性的框架，提出一种泛化目标 HUG，通过测量 GNC 和 HUG 之间的差异来解耦 GNC，作为一种有效的交叉墒代价函数的替代函数。

通过超球形均匀间隙实现神经坍塌的泛化和解耦

Generalizing and Decoupling Neural Collapse via Hyperspherical  Uniformity Gap

Due to the over-parameterization nature, neural networks are a powerful tool
for nonlinear function approximation. In order to achieve good generalization
on unseen data, a suitable inductive bias is of great importance for neural
networks. One of the most straightforward ways is to regularize the neural
network with some additional objectives. L2 regularization serves as a standard
regularization for neural networks. Despite its popularity, it essentially
regularizes one dimension of the individual neuron, which is not strong enough
to control the capacity of highly over-parameterized neural networks. Motivated
by this, hyperspherical uniformity is proposed as a novel family of relational
regularizations that impact the interaction among neurons. We consider several
geometrically distinct ways to achieve hyperspherical uniformity. The
effectiveness of hyperspherical uniformity is justified by theoretical insights
and empirical evaluations.

本文提出了一种新的关系规范化方法 —— 超球均一性，以影响神经元之间的相互作用，进而为神经网络打下适当的归纳偏差，从而实现对新数据的良好泛化。