Recently, Transformers have emerged as the go-to architecture for both vision
and language modeling tasks, but their computational efficiency is limited by
the length of the input sequence. To address this, several efficient variants
of Transformers have been proposed to accelerate computation or reduce memory
consumption while preserving performance. This paper presents an efficient
vision Transformer, called CageViT, that is guided by convolutional activation
to reduce computation. Our CageViT, unlike current Transformers, utilizes a new
encoder to handle the rearranged tokens, bringing several technical
contributions: 1) Convolutional activation is used to pre-process the token
after patchifying the image to select and rearrange the major tokens and minor
tokens, which substantially reduces the computation cost through an additional
fusion layer. 2) Instead of using the class activation map of the convolutional
model directly, we design a new weighted class activation to lower the model
requirements. 3) To facilitate communication between major tokens and fusion
tokens, Gated Linear SRA is proposed to further integrate fusion tokens into
the attention mechanism. We perform a comprehensive validation of CageViT on
the image classification challenge.
Experimental results demonstrate that the proposed CageViT outperforms the
most recent state-of-the-art backbones by a large margin in terms of
efficiency, while maintaining a comparable level of accuracy (e.g. a
moderate-sized 43.35M model trained solely on 224 x 224 ImageNet-1K can achieve
Top-1 accuracy of 83.4% accuracy).

本文提出了一种新的有效的神经网络结构 CageViT，该结构利用卷积激活来减少计算，并通过新的编码器来处理重组的标记；实验结果表明，CageViT 在图像分类挑战中的有效性优于当前最新的先进骨干。