Recent developments in Multimodal Large Language Models (MLLMs) have shown
rapid progress, moving towards the goal of creating versatile MLLMs that
understand inputs from various modalities. However, existing methods typically
rely on joint training with paired multimodal instruction data, which is
resource-intensive and challenging to extend to new modalities. In this paper,
we propose a new paradigm through the model composition of existing MLLMs to
create a new model that retains the modal understanding capabilities of each
original model. Our basic implementation, NaiveMC, demonstrates the
effectiveness of this paradigm by reusing modality encoders and merging LLM
parameters. Furthermore, we introduce DAMC to address parameter interference
and mismatch issues during the merging process, thereby enhancing the model
performance. To facilitate research in this area, we propose MCUB, a benchmark
for assessing ability of MLLMs to understand inputs from diverse modalities.
Experiments on this benchmark and four other multimodal understanding tasks
show significant improvements over baselines, proving that model composition
can create a versatile model capable of processing inputs from multiple
modalities.

在该篇论文中，研究人员介绍了一种新的范例通过现有 MLLMs 的模型组合来创建一个新模型，该模型保留了每个原始模型的模态理解能力。通过实验证明，模型组合可以创建一个多功能模型，能够处理来自多种模态的输入数据。