Recently, a vast amount of literature has focused on the "Neural Collapse"
(NC) phenomenon, which emerges when training neural network (NN) classifiers
beyond the zero training error point. The core component of NC is the decrease
in the within class variability of the network's deepest features, dubbed as
NC1. The theoretical works that study NC are typically based on simplified
unconstrained features models (UFMs) that mask any effect of the data on the
extent of collapse. In this paper, we provide a kernel-based analysis that does
not suffer from this limitation. First, given a kernel function, we establish
expressions for the traces of the within- and between-class covariance matrices
of the samples' features (and consequently an NC1 metric). Then, we turn to
focus on kernels associated with shallow NNs. First, we consider the NN
Gaussian Process kernel (NNGP), associated with the network at initialization,
and the complement Neural Tangent Kernel (NTK), associated with its training in
the "lazy regime". Interestingly, we show that the NTK does not represent more
collapsed features than the NNGP for prototypical data models. As NC emerges
from training, we then consider an alternative to NTK: the recently proposed
adaptive kernel, which generalizes NNGP to model the feature mapping learned
from the training data. Contrasting our NC1 analysis for these two kernels
enables gaining insights into the effect of data distribution on the extent of
collapse, which are empirically aligned with the behavior observed with
practical training of NNs.

最近，大量文献集中在 “神经坍塌”（NC）现象上，当训练神经网络分类器超过零误差训练点时出现。NC 的核心组成部分是网络最深特征的类内变异性的减少，被称为 NC1。我们提供了一个基于核的分析，不受数据对坍塌程度的影响的简化非约束特征模型（UFM）的限制。通过给定核函数，建立样本特征的类内和类间协方差矩阵的迹的表达式（因此是 NC1 度量）。然后，我们转向研究与浅层神经网络相关的核。首先，我们考虑神经网络高斯过程核（NNGP），与初始化时的网络相关联，并且补充神经切线核（NTK），与其在 “懒惰模式” 下的训练相关。有趣的是，我们证明对于典型数据模型，NTK 并不表示比 NNGP 更多坍塌的特征。当 NC 出现时，我们考虑一个替代 NTK 的方法：最近提出的自适应核，它将 NNGP 推广为从训练数据中学到的特征映射模型。通过对这两种核的 NC1 分析进行对比，可以获得关于数据分布对坍塌程度的影响的见解，这与实际训练神经网络的行为符合实证观察。

核心与核心：探索数据结构如何影响神经崩溃

Kernel vs. Kernel: Exploring How the Data Structure Affects Neural  Collapse

Efficient optimisation of black-box problems that comprise both continuous
and categorical inputs is important, yet poses significant challenges. We
propose a new approach, Continuous and Categorical Bayesian Optimisation
(CoCaBO), which combines the strengths of multi-armed bandits and Bayesian
optimisation to select values for both categorical and continuous inputs. We
model this mixed-type space using a Gaussian Process kernel, designed to allow
sharing of information across multiple categorical variables, each with
multiple possible values; this allows CoCaBO to leverage all available data
efficiently. We extend our method to the batch setting and propose an efficient
selection procedure that dynamically balances exploration and exploitation
whilst encouraging batch diversity. We demonstrate empirically that our method
outperforms existing approaches on both synthetic and real-world optimisation
tasks with continuous and categorical inputs.

提出了一种基于混合类型空间的新方法 CoCaBO，它结合了多臂赌博机和贝叶斯优化的优点，使用高斯过程核来对包含连续和分类输入的黑盒问题进行优化，实现了在探索和开发之间的平衡，并在连续和分类输入的合成和实际优化任务上超越了现有方法。