While models derived from Vision Transformers (ViTs) have been phonemically
surging, pre-trained models cannot seamlessly adapt to arbitrary resolution
images without altering the architecture and configuration, such as sampling
the positional encoding, limiting their flexibility for various vision tasks.
For instance, the Segment Anything Model (SAM) based on ViT-Huge requires all
input images to be resized to 1024$\times$1024. To overcome this limitation, we
propose the Multi-Head Self-Attention Convolution (MSA-Conv) that incorporates
Self-Attention within generalized convolutions, including standard, dilated,
and depthwise ones. Enabling transformers to handle images of varying sizes
without retraining or rescaling, the use of MSA-Conv further reduces
computational costs compared to global attention in ViT, which grows costly as
image size increases. Later, we present the Vision Transformer in Convolution
(TiC) as a proof of concept for image classification with MSA-Conv, where two
capacity enhancing strategies, namely Multi-Directional Cyclic Shifted
Mechanism and Inter-Pooling Mechanism, have been proposed, through establishing
long-distance connections between tokens and enlarging the effective receptive
field. Extensive experiments have been carried out to validate the overall
effectiveness of TiC. Additionally, ablation studies confirm the performance
improvement made by MSA-Conv and the two capacity enhancing strategies
separately. Note that our proposal aims at studying an alternative to the
global attention used in ViT, while MSA-Conv meets our goal by making TiC
comparable to state-of-the-art on ImageNet-1K. Code will be released at
this https URL

通过结合 Self-Attention 与广义卷积（包括标准卷积、空洞卷积和深度卷积），Multi-Head Self-Attention Convolution (MSA-Conv) 扩展了 Transformer 模型，使其能够处理不同尺寸的图像，降低了计算成本，并且通过引入两种增强策略实现了长距离连接和扩大的感受野，而在图像分类任务中，Vision Transformer in Convolution (TiC) 利用了 MSA-Conv，在 ImageNet-1K 数据集上与目前最先进的方法具有可比性。