Recent works have shown that transformers can solve contextual reasoning
tasks by internally executing computational graphs called circuits. Circuits
often use attention to logically match information from subspaces of the
representation, e.g. using position-in-sequence to identify the previous token.
In this work, we consider a semantic subspace to be any independent subspace of
the latent representation that can fully determine an attention distribution.
We show that Pre-Norm, the placement of normalisation layer used by
state-of-the-art transformers, violates this ability unless the model learns a
strict representation structure of orthogonal spheres. This is because it
causes linear subspaces to interfere through their common normalisation factor.
Theoretically, we analyse circuit stability by modelling this interference as
random noise on the $L_2$-norms of the query/key/value vectors, predicting a
phenomenon of circuit collapse when sparse-attention shifts to a different
token. Empirically, we investigate the sensitivity of real-world models trained
for mathematical addition, observing a 1% rate of circuit collapse when the
norms are artificially perturbed by $\lesssim$10%. We contrast Pre-Norm with
QKV-Norm, which places normalisation after the attention head's linear
operators. Theoretically this relaxes the representational constraints.
Empirically we observe comparable in-distribution but worse out-of-distribution
performance.

传统的预归一化（Pre-Norm）方法在 transformer 模型中引入线性子空间干扰，导致电路塌陷现象；相比之下，将归一化操作放在注意力头的线性操作之后（QKV-Norm）能更好地保持表示的稳定性。

Transformer 规范化层与语义子空间的独立性

Transformer Normalisation Layers and the Independence of Semantic  Subspaces

Deep kernel machines (DKMs) are a recently introduced kernel method with the
flexibility of other deep models including deep NNs and deep Gaussian
processes. DKMs work purely with kernels, never with features, and are
therefore different from other methods ranging from NNs to deep kernel learning
and even deep Gaussian processes, which all use features as a fundamental
component. Here, we introduce convolutional DKMs, along with an efficient
inter-domain inducing point approximation scheme. Further, we develop and
experimentally assess a number of model variants, including 9 different types
of normalisation designed for the convolutional DKMs, two likelihoods, and two
different types of top-layer. The resulting models achieve around 99% test
accuracy on MNIST, 92% on CIFAR-10 and 71% on CIFAR-100, despite training in
only around 28 GPU hours, 1-2 orders of magnitude faster than full NNGP / NTK /
Myrtle kernels, whilst achieving comparable performance.

深度核机器（DKMs）是一种最近引入的核方法，具有包括深度神经网络和深高斯过程在内的其他深度模型的灵活性。本文介绍了卷积 DKMs 以及一种高效的跨域引导点逼近方案，并且开发和实验评估了多种模型变体，包括针对卷积 DKMs 设计的 9 种不同类型的规范化方法、两种似然函数和两种不同类型的顶层。尽管训练时间只需要大约 28 个 GPU 小时，比完整的 NNGP/NTK/Myrtle 核函数快 1 到 2 个数量级，但结果模型在 MNIST 上达到约 99% 的测试准确率，在 CIFAR-10 上达到 92% 的准确率，在 CIFAR-100 上达到 71% 的准确率，同时实现了可比较的性能。

卷积深度核机器

Convolutional Deep Kernel Machines

In this paper we revisit the efficacy of knowledge distillation as a function
matching and metric learning problem. In doing so we verify three important
design decisions, namely the normalisation, soft maximum function, and
projection layers as key ingredients. We theoretically show that the projector
implicitly encodes information on past examples, enabling relational gradients
for the student. We then show that the normalisation of representations is
tightly coupled with the training dynamics of this projector, which can have a
large impact on the students performance. Finally, we show that a simple soft
maximum function can be used to address any significant capacity gap problems.
Experimental results on various benchmark datasets demonstrate that using these
insights can lead to superior or comparable performance to state-of-the-art
knowledge distillation techniques, despite being much more computationally
efficient. In particular, we obtain these results across image classification
(CIFAR100 and ImageNet), object detection (COCO2017), and on more difficult
distillation objectives, such as training data efficient transformers, whereby
we attain a 77.2% top-1 accuracy with DeiT-Ti on ImageNet.

本文通过实验证明了正则化 (即 normalisation)、软最大值函数以及投影层是知识蒸馏的关键因素，并提出了一种简单的软最大函数来解决容量差异问题。实验结果表明，使用这些洞见可以实现与最先进的知识蒸馏技术相当或更好的性能，而且计算效率更高。

知识蒸馏训练动态深入研究

A closer look at the training dynamics of knowledge distillation

Divergence from a random baseline is a technique for the evaluation of
document clustering. It ensures cluster quality measures are performing work
that prevents ineffective clusterings from giving high scores to clusterings
that provide no useful result. These concepts are defined and analysed using
intrinsic and extrinsic approaches to the evaluation of document cluster
quality. This includes the classical clusters to categories approach and a
novel approach that uses ad hoc information retrieval. The divergence from a
random baseline approach is able to differentiate ineffective clusterings
encountered in the INEX XML Mining track. It also appears to perform a
normalisation similar to the Normalised Mutual Information (NMI) measure but it
can be applied to any measure of cluster quality. When it is applied to the
intrinsic measure of distortion as measured by RMSE, subtraction from a random
baseline provides a clear optimum that is not apparent otherwise. This approach
can be applied to any clustering evaluation. This paper describes its use in
the context of document clustering evaluation.

本文介绍了一种利用随机基线进行文档聚类评估的方法，可以区分不合适的聚类并对聚类质量指标进行归一化处理，可用于任何聚类评估。