Following their success in natural language processing, transformers have recently shown much promise for computer vision. The self-attention operation underlying transformers yields global interactions between all tokens ,i.e. words or image patches, and enables flexible modelling of image data beyond the local interactions of convolutions. This flexibility, however, comes with a quadratic complexity in time and memory, hindering application to long sequences and high-resolution images. We propose a "transposed" version of self-attention that operates across feature channels rather than tokens, where the interactions are based on the cross-covariance matrix between keys and queries. The resulting cross-covariance attention (XCA) has linear complexity in the number of tokens, and allows efficient processing of high-resolution images. Our cross-covariance image transformer (XCiT) is built upon XCA. It combines the accuracy of conventional transformers with the scalability of convolutional architectures. We validate the effectiveness and generality of XCiT by reporting excellent results on multiple vision benchmarks, including image classification and self-supervised feature learning on ImageNet-1k, object detection and instance segmentation on COCO, and semantic segmentation on ADE20k.

本文介绍了基于交叉协方差矩阵的交叉协方差注意力（XCA），用于高分辨率图像的高效处理。文章基于XCA构建了交叉协方差图像变换器（XCiT），并在多个视觉基准测试中取得了优异的结果，包括ImageNet-1k上的图像分类和自监督特征学习，COCO上的目标检测和实例分割以及ADE20K上的语义分割。

XCiT: 跨协方差图像变换器