The remarkable success of transformers in sequence modeling tasks, spanning
various applications in natural language processing and computer vision, is
attributed to the critical role of self-attention. Similar to the development
of most deep learning models, the construction of these attention mechanisms
rely on heuristics and experience. In our work, we derive self-attention from
kernel principal component analysis (kernel PCA) and show that self-attention
projects its query vectors onto the principal component axes of its key matrix
in a feature space. We then formulate the exact formula for the value matrix in
self-attention, theoretically and empirically demonstrating that this value
matrix captures the eigenvectors of the Gram matrix of the key vectors in
self-attention. Leveraging our kernel PCA framework, we propose Attention with
Robust Principal Components (RPC-Attention), a novel class of robust attention
that is resilient to data contamination. We empirically demonstrate the
advantages of RPC-Attention over softmax attention on the ImageNet-1K object
classification, WikiText-103 language modeling, and ADE20K image segmentation
task.

我们通过核主成分分析推导了自注意力机制，表明自注意力将其查询向量投影到特征空间中其关键矩阵的主成分轴上。利用我们的核主成分分析框架，我们提出了一种鲁棒的自注意力机制 RPC-Attention，该机制对数据污染具有弹性，并在 ImageNet-1K 目标分类、WikiText-103 语言建模和 ADE20K 图像分割任务中通过实验证明了 RPC-Attention 相对于 softmax attention 的优势。

通过核主成分分析揭示自注意力的隐藏结构

Unveiling the Hidden Structure of Self-Attention via Kernel Principal  Component Analysis

Out-of-Distribution (OoD) detection is vital for the reliability of Deep
Neural Networks (DNNs). Existing works have shown the insufficiency of
Principal Component Analysis (PCA) straightforwardly applied on the features of
DNNs in detecting OoD data from In-Distribution (InD) data. The failure of PCA
suggests that the network features residing in OoD and InD are not well
separated by simply proceeding in a linear subspace, which instead can be
resolved through proper nonlinear mappings. In this work, we leverage the
framework of Kernel PCA (KPCA) for OoD detection, seeking subspaces where OoD
and InD features are allocated with significantly different patterns. We devise
two feature mappings that induce non-linear kernels in KPCA to advocate the
separability between InD and OoD data in the subspace spanned by the principal
components. Given any test sample, the reconstruction error in such subspace is
then used to efficiently obtain the detection result with $\mathcal{O}(1)$ time
complexity in inference. Extensive empirical results on multiple OoD data sets
and network structures verify the superiority of our KPCA-based detector in
efficiency and efficacy with state-of-the-art OoD detection performances.

利用核主成分分析（KPCA）框架进行异常数据检测，通过非线性核映射来提高内分布和外分布特征的分离性能，实验证明其在效率和性能方面优于现有的最先进的异常数据检测方法。

用于异常检测的核主成分分析

Kernel PCA for Out-of-Distribution Detection

In contrast to deep networks, kernel methods cannot directly take advantage
of depth. In this regard, the deep Restricted Kernel Machine (DRKM) framework
allows multiple levels of kernel PCA (KPCA) and Least-Squares Support Vector
Machines (LSSVM) to be combined into a deep architecture using visible and
hidden units. We propose a new method for DRKM classification coupling the
objectives of KPCA and classification levels, with the hidden feature matrix
lying on the Stiefel manifold. The classification level can be formulated as an
LSSVM or as an MLP feature map, combining depth in terms of levels and layers.
The classification level is expressed in its primal formulation, as the deep
KPCA levels can embed the most informative components of the data in a much
lower dimensional space. In the experiments on benchmark datasets with few
available training points, we show that our deep method improves over the
LSSVM/MLP and that models with multiple KPCA levels can outperform models with
a single level.

本研究提出了一种基于深度受限核机 (DRKM) 框架的新的分类方法，将 KPCA 和分类级别的目标结合起来，其中隐藏特征矩阵位于 Stiefel 流形上，分类级别可以是 LSSVM 或 MLP 特征映射，深 KPCA 级别可以嵌入数据的最具信息组件。在基准数据集上的实验表明，我们的深度方法优于 LSSVM/MLP，并且具有多个 KPCA 级别的模型可优于具有单个级别的模型。