The current paradigm of training deep neural networks for classification
tasks includes minimizing the empirical risk that pushes the training loss
value towards zero, even after the training error has been vanished. In this
terminal phase of training, it has been observed that the last-layer features
collapse to their class-means and these class-means converge to the vertices of
a simplex Equiangular Tight Frame (ETF). This phenomenon is termed as Neural
Collapse (NC). To theoretically understand this phenomenon, recent works employ
a simplified unconstrained feature model to prove that NC emerges at the global
solutions of the training problem. However, when the training dataset is
class-imbalanced, some NC properties will no longer be true. For example, the
class-means geometry will skew away from the simplex ETF when the loss
converges. In this paper, we generalize NC to imbalanced regime for
cross-entropy loss under the unconstrained ReLU feature model. We prove that,
while the within-class features collapse property still holds in this setting,
the class-means will converge to a structure consisting of orthogonal vectors
with different lengths. Furthermore, we find that the classifier weights are
aligned to the scaled and centered class-means with scaling factors depend on
the number of training samples of each class, which generalizes NC in the
class-balanced setting. We empirically prove our results through experiments on
practical architectures and dataset.

在该研究中，我们将神经坍缩现象推广到不平衡类别的交叉熵损失下，证明了虽然类内特征会坍缩，但类均值将收敛到长度不同的正交向量结构，并发现分类器权重与类均值的缩放和居中相关，这扩展了在平衡类别设置下的神经坍缩现象。通过在实际架构和数据集上进行实验证明了我们的结果。

基于无约束 ReLU 特征模型的交叉熵类不平衡学习的神经倒塌

Neural Collapse for Cross-entropy Class-Imbalanced Learning with  Unconstrained ReLU Feature Model

Recent deep networks achieved state of the art performance on a variety of
semantic segmentation tasks. Despite such progress, these models often face
challenges in real world `wild tasks' where large difference between labeled
training/source data and unseen test/target data exists. In particular, such
difference is often referred to as `domain gap', and could cause significantly
decreased performance which cannot be easily remedied by further increasing the
representation power. Unsupervised domain adaptation (UDA) seeks to overcome
such problem without target domain labels. In this paper, we propose a novel
UDA framework based on an iterative self-training procedure, where the problem
is formulated as latent variable loss minimization, and can be solved by
alternatively generating pseudo labels on target data and re-training the model
with these labels. On top of self-training, we also propose a novel
class-balanced self-training framework to avoid the gradual dominance of large
classes on pseudo-label generation, and introduce spatial priors to refine
generated labels. Comprehensive experiments show that the proposed methods
achieve state of the art semantic segmentation performance under multiple major
UDA settings.

本文提出了一种基于迭代自训练的无监督域自适应框架，在自训练的基础上提出了一种新的类平衡自训练框架，并引入了空间先验来改善生成的标签。综合实验表明，所提出的方法在多个主要 UDA 设置下均实现了最先进的语义分割性能。