Recently the state space models (SSMs) with efficient hardware-aware designs,
i.e., Mamba, have shown great potential for long sequence modeling. Building
efficient and generic vision backbones purely upon SSMs is an appealing
direction. However, representing visual data is challenging for SSMs due to the
position-sensitivity of visual data and the requirement of global context for
visual understanding. In this paper, we show that the reliance of visual
representation learning on self-attention is not necessary and propose a new
generic vision backbone with bidirectional Mamba blocks (Vim), which marks the
image sequences with position embeddings and compresses the visual
representation with bidirectional state space models. On ImageNet
classification, COCO object detection, and ADE20k semantic segmentation tasks,
Vim achieves higher performance compared to well-established vision
transformers like DeiT, while also demonstrating significantly improved
computation & memory efficiency. For example, Vim is 2.8$\times$ faster than
DeiT and saves 86.8% GPU memory when performing batch inference to extract
features on images with a resolution of 1248$\times$1248. The results
demonstrate that Vim is capable of overcoming the computation & memory
constraints on performing Transformer-style understanding for high-resolution
images and it has great potential to become the next-generation backbone for
vision foundation models. Code is available at this https URL

本文提出了一种新的通用计算机视觉基础模型 Vim，该模型采用双向状态空间模型对图像序列进行标记并压缩视觉表示，并在 ImageNet 分类、COCO 目标检测和 ADE20k 语义分割任务中取得了比 DeiT 等常见视觉转换器更高的性能，同时具有显著提高的计算和内存效率。