Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs) stand as
the two most popular foundation models for visual representation learning.
While CNNs exhibit remarkable scalability with linear complexity w.r.t. image
resolution, ViTs surpass them in fitting capabilities despite contending with
quadratic complexity. A closer inspection reveals that ViTs achieve superior
visual modeling performance through the incorporation of global receptive
fields and dynamic weights. This observation motivates us to propose a novel
architecture that inherits these components while enhancing computational
efficiency. To this end, we draw inspiration from the recently introduced state
space model and propose the Visual State Space Model (VMamba), which achieves
linear complexity without sacrificing global receptive fields. To address the
encountered direction-sensitive issue, we introduce the Cross-Scan Module (CSM)
to traverse the spatial domain and convert any non-causal visual image into
order patch sequences. Extensive experimental results substantiate that VMamba
not only demonstrates promising capabilities across various visual perception
tasks, but also exhibits more pronounced advantages over established benchmarks
as the image resolution increases. Source code has been available at
this https URL

基于卷积神经网络和视觉变换器，提出了一种名为 Visual State Space Model (VMamba) 的新型体系结构，具有线性复杂度和全局感受野，并在图像分辨率增加时展现出更明显的优势。