Despite the significant achievements of Vision Transformers (ViTs) in various
vision tasks, they are constrained by the quadratic complexity. Recently, State
Space Models (SSMs) have garnered widespread attention due to their global
receptive field and linear complexity with respect to the input length,
demonstrating substantial potential across fields including natural language
processing and computer vision. To improve the performance of SSMs in vision
tasks, a multi-scan strategy is widely adopted, which leads to significant
redundancy of SSMs. For a better trade-off between efficiency and performance,
we analyze the underlying reasons behind the success of the multi-scan
strategy, where long-range dependency plays an important role. Based on the
analysis, we introduce Multi-Scale Vision Mamba (MSVMamba) to preserve the
superiority of SSMs in vision tasks with limited parameters. It employs a
multi-scale 2D scanning technique on both original and downsampled feature
maps, which not only benefits long-range dependency learning but also reduces
computational costs. Additionally, we integrate a Convolutional Feed-Forward
Network (ConvFFN) to address the lack of channel mixing. Our experiments
demonstrate that MSVMamba is highly competitive, with the MSVMamba-Tiny model
achieving 82.8% top-1 accuracy on ImageNet, 46.9% box mAP, and 42.2% instance
mAP with the Mask R-CNN framework, 1x training schedule on COCO, and 47.6% mIoU
with single-scale testing on ADE20K.Code is available at
https://github.com/YuHengsss/MSVMamba.

通过分析多次扫描策略在多尺度视觉任务中的成功原因，本研究介绍了 Multi-Scale Vision Mamba（MSVMamba）模型，采用多尺度二维扫描技术并整合了卷积前馈网络（ConvFFN），在有限参数下保留了 State Space Models（SSMs）在视觉任务中的卓越性能。