Online continual learning suffers from an underfitted solution due to
insufficient training for prompt model update (e.g., single-epoch training). To
address the challenge, we propose an efficient online continual learning method
using the neural collapse phenomenon. In particular, we induce neural collapse
to form a simplex equiangular tight frame (ETF) structure in the representation
space so that the continuously learned model with a single epoch can better fit
to the streamed data by proposing preparatory data training and residual
correction in the representation space. With an extensive set of empirical
validations using CIFAR-10/100, TinyImageNet, ImageNet-200, and ImageNet-1K, we
show that our proposed method outperforms state-of-the-art methods by a
noticeable margin in various online continual learning scenarios such as
disjoint and Gaussian scheduled continuous (i.e., boundary-free) data setups.

我们提出了一种利用神经坍塌现象的高效在线持续学习方法，通过在表示空间中进行预备数据训练和残差修正，诱导神经坍塌形成简单光角紧框架（ETF）结构，使得连续学习的模型在仅进行单个时期的训练后更好地适应流数据。在使用 CIFAR-10/100、TinyImageNet、ImageNet-200 和 ImageNet-1K 进行了大量实证验证后，我们证明了我们提出的方法在各种在线持续学习场景中（如不相交和高斯调度连续（即无边界）数据设置）明显优于现有方法。

在线持续学习的学习等角表示

Learning Equi-angular Representations for Online Continual Learning

The recently discovered Neural collapse (NC) phenomenon states that the
last-layer weights of Deep Neural Networks (DNN), converge to the so-called
Equiangular Tight Frame (ETF) simplex, at the terminal phase of their training.
This ETF geometry is equivalent to vanishing within-class variability of the
last layer activations. Inspired by NC properties, we explore in this paper the
transferability of DNN models trained with their last layer weight fixed
according to ETF. This enforces class separation by eliminating class
covariance information, effectively providing implicit regularization. We show
that DNN models trained with such a fixed classifier significantly improve
transfer performance, particularly on out-of-domain datasets. On a broad range
of fine-grained image classification datasets, our approach outperforms i)
baseline methods that do not perform any covariance regularization (up to 22%),
as well as ii) methods that explicitly whiten covariance of activations
throughout training (up to 19%). Our findings suggest that DNNs trained with
fixed ETF classifiers offer a powerful mechanism for improving transfer
learning across domains.

通过固定 ETF 权重的 DNN 模型训练，我们展示了在跨域数据集上显著提高传输性能的能力，尤其是在细粒度图像分类数据集上，超过了基准方法和明确漂白活跃度协方差的方法。

使用固定随机分类器训练的深度神经网络模型在领域间转移更好

Deep Neural Network Models Trained With A Fixed Random Classifier  Transfer Better Across Domains

There is a recently discovered and intriguing phenomenon called Neural
Collapse: at the terminal phase of training a deep neural network for
classification, the within-class penultimate feature means and the associated
classifier vectors of all flat classes collapse to the vertices of a simplex
Equiangular Tight Frame (ETF). Recent work has tried to exploit this phenomenon
by fixing the related classifier weights to a pre-computed ETF to induce neural
collapse and maximize the separation of the learned features when training with
imbalanced data. In this work, we propose to fix the linear classifier of a
deep neural network to a Hierarchy-Aware Frame (HAFrame), instead of an ETF,
and use a cosine similarity-based auxiliary loss to learn hierarchy-aware
penultimate features that collapse to the HAFrame. We demonstrate that our
approach reduces the mistake severity of the model's predictions while
maintaining its top-1 accuracy on several datasets of varying scales with
hierarchies of heights ranging from 3 to 12. We will release our code on GitHub
in the near future.

该研究提出了一种新的深度神经网络架构，通过固定线性分类器到 Hierarchy-Aware Frame (HAFrame)，利用余弦相似度辅助损失函数来学习 collapse 到 HAFrame 的层级感知特征，从而降低模型预测的误差且保持其准确性。