Recently, a vast amount of literature has focused on the "Neural Collapse"
(NC) phenomenon, which emerges when training neural network (NN) classifiers
beyond the zero training error point. The core component of NC is the decrease
in the within class variability of the network's deepest features, dubbed as
NC1. The theoretical works that study NC are typically based on simplified
unconstrained features models (UFMs) that mask any effect of the data on the
extent of collapse. In this paper, we provide a kernel-based analysis that does
not suffer from this limitation. First, given a kernel function, we establish
expressions for the traces of the within- and between-class covariance matrices
of the samples' features (and consequently an NC1 metric). Then, we turn to
focus on kernels associated with shallow NNs. First, we consider the NN
Gaussian Process kernel (NNGP), associated with the network at initialization,
and the complement Neural Tangent Kernel (NTK), associated with its training in
the "lazy regime". Interestingly, we show that the NTK does not represent more
collapsed features than the NNGP for prototypical data models. As NC emerges
from training, we then consider an alternative to NTK: the recently proposed
adaptive kernel, which generalizes NNGP to model the feature mapping learned
from the training data. Contrasting our NC1 analysis for these two kernels
enables gaining insights into the effect of data distribution on the extent of
collapse, which are empirically aligned with the behavior observed with
practical training of NNs.

最近，大量文献集中在 “神经坍塌”（NC）现象上，当训练神经网络分类器超过零误差训练点时出现。NC 的核心组成部分是网络最深特征的类内变异性的减少，被称为 NC1。我们提供了一个基于核的分析，不受数据对坍塌程度的影响的简化非约束特征模型（UFM）的限制。通过给定核函数，建立样本特征的类内和类间协方差矩阵的迹的表达式（因此是 NC1 度量）。然后，我们转向研究与浅层神经网络相关的核。首先，我们考虑神经网络高斯过程核（NNGP），与初始化时的网络相关联，并且补充神经切线核（NTK），与其在 “懒惰模式” 下的训练相关。有趣的是，我们证明对于典型数据模型，NTK 并不表示比 NNGP 更多坍塌的特征。当 NC 出现时，我们考虑一个替代 NTK 的方法：最近提出的自适应核，它将 NNGP 推广为从训练数据中学到的特征映射模型。通过对这两种核的 NC1 分析进行对比，可以获得关于数据分布对坍塌程度的影响的见解，这与实际训练神经网络的行为符合实证观察。