Vision transformers (ViTs) have found only limited practical use in
processing images, in spite of their state-of-the-art accuracy on certain
benchmarks. The reason for their limited use include their need for larger
training datasets and more computational resources compared to convolutional
neural networks (CNNs), owing to the quadratic complexity of their
self-attention mechanism. We propose a linear attention-convolution hybrid
architecture -- Convolutional X-formers for Vision (CXV) -- to overcome these
limitations. We replace the quadratic attention with linear attention
mechanisms, such as Performer, Nyströmformer, and Linear Transformer, to
reduce its GPU usage. Inductive prior for image data is provided by
convolutional sub-layers, thereby eliminating the need for class token and
positional embeddings used by the ViTs. We also propose a new training method
where we use two different optimizers during different phases of training and
show that it improves the top-1 image classification accuracy across different
architectures. CXV outperforms other architectures, token mixers (e.g.
ConvMixer, FNet and MLP Mixer), transformer models (e.g. ViT, CCT, CvT and
hybrid Xformers), and ResNets for image classification in scenarios with
limited data and GPU resources (cores, RAM, power).

提出了一种基于线性注意力机制的混合体系结构 ——Convolutional X-formers for Vision（CXV）。通过将 Quintic Transformer，Nyströmformer 和 Linear Transformer 等线性注意力机制代替二次注意力机制，来减少 GPU 使用。CXV 在有限的数据和 GPU 资源（核心，内存，功率）场景下，比其他的架构如 Token mixers（例如 ConvMixer，Fnet 和 MLP Mixer），变换模型（如 ViT，CCT，CvT 和混合 Xformers）以及 ResNets 等，更适用于图像分类任务。