In the past year, MultiModal Large Language Models (MM-LLMs) have undergone substantial advancements, augmenting off-the-shelf LLMs to support MM inputs or outputs via cost-effective training strategies. The resulting models not only preserve the inherent reasoning and decision-making capabilities of LLMs but also empower a diverse range of MM tasks. In this paper, we provide a comprehensive survey aimed at facilitating further research of MM-LLMs. Specifically, we first outline general design formulations for model architecture and training pipeline. Subsequently, we provide brief introductions of $26$ existing MM-LLMs, each characterized by its specific formulations. Additionally, we review the performance of MM-LLMs on mainstream benchmarks and summarize key training recipes to enhance the potency of MM-LLMs. Lastly, we explore promising directions for MM-LLMs while concurrently maintaining a real-time tracking website for the latest developments in the field. We hope that this survey contributes to the ongoing advancement of the MM-LLMs domain.

综述了过去一年多模态大型语言模型（MM-LLMs）的进展，提供了对MM-LLMs的全面调查，包括模型架构和训练流程的设计概述，介绍了26种不同形式的MM-LLMs，并回顾了其在主流基准上的性能以及提升其效力的关键训练方法，同时探索了MM-LLMs领域的前景方向。

多模态大语言模型（MM-LLMs）的最新进展