Multimodal Large Language Models (MLLMs) have showcased impressive skills in tasks related to visual understanding and reasoning. Yet, their widespread application faces obstacles due to the high computational demands during both the training and inference phases, restricting their use to a limited audience within the research and user communities. In this paper, we investigate the design aspects of Multimodal Small Language Models (MSLMs) and propose an efficient multimodal assistant named Mipha, which is designed to create synergy among various aspects: visual representation, language models, and optimization strategies. We show that without increasing the volume of training data, our Mipha-3B outperforms the state-of-the-art large MLLMs, especially LLaVA-1.5-13B, on multiple benchmarks. Through detailed discussion, we provide insights and guidelines for developing strong MSLMs that rival the capabilities of MLLMs. Our code is available at https://github.com/zhuyiche/Mipha.

在本文中，我们研究了Multimodal Small Language Models（MSLMs）的设计方面，并提出了一种高效的多模态助手Mipha，旨在在视觉表示、语言模型和优化策略等各个方面产生协同效应。我们展示了在不增加训练数据量的情况下，我们的Mipha-3B在多个基准测试中表现优于最先进的大型MLLMs，尤其是LLaVA-1.5-13B。通过详细讨论，我们提供了开发强大的MSLMs以与MLLMs能力匹敌的见解和指南。

小语言模型的多模态助手全面重构