In recent years, the application of multimodal large language models (MLLM)
in various fields has achieved remarkable success. However, as the foundation
model for many downstream tasks, current MLLMs are composed of the well-known
Transformer network, which has a less efficient quadratic computation
complexity. To improve the efficiency of such basic models, we propose Cobra, a
linear computational complexity MLLM. Specifically, Cobra integrates the
efficient Mamba language model into the visual modality. Moreover, we explore
and study various modal fusion schemes to create an effective multi-modal
Mamba. Extensive experiments demonstrate that (1) Cobra achieves extremely
competitive performance with current computationally efficient state-of-the-art
methods, \textit{e.g.}, LLaVA-Phi, TinyLLaVA, and MobileVLM v2, and has faster
speed due to Cobra's linear sequential modeling. (2) Interestingly, the results
of closed-set challenging prediction benchmarks show that Cobra performs well
in overcoming visual illusions and spatial relationship judgments. (3) Notably,
Cobra even achieves comparable performance to LLaVA with about 43% of the
number of parameters. We will make all codes of Cobra open-source and hope that
the proposed method can facilitate future research on complexity problems in
MLLM. Our project page is available at: this https URL

通过将高效的 Mamba 语言模型引入视觉模态，Cobra 实现了线性计算复杂度的多模态大型语言模型，其在现有的计算效率高的方法上表现出极具竞争力的性能，并且具有更快的速度；同时，Cobra 在克服视觉错觉和空间关系判断方面表现出色，甚至与 LLaVA 在参数数量只有 43% 时表现相当；我们将开源 Cobra 的所有代码，希望该方法能促进 MLLM 的复杂性问题的未来研究。