Recent developments in Multimodal Large Language Models (MLLMs) have shown
rapid progress, moving towards the goal of creating versatile MLLMs that
understand inputs from various modalities. However, existing methods typically
rely on joint training with paired multimodal instruction data, which is
resource-intensive and challenging to extend to new modalities. In this paper,
we propose a new paradigm through the model composition of existing MLLMs to
create a new model that retains the modal understanding capabilities of each
original model. Our basic implementation, NaiveMC, demonstrates the
effectiveness of this paradigm by reusing modality encoders and merging LLM
parameters. Furthermore, we introduce DAMC to address parameter interference
and mismatch issues during the merging process, thereby enhancing the model
performance. To facilitate research in this area, we propose MCUB, a benchmark
for assessing ability of MLLMs to understand inputs from diverse modalities.
Experiments on this benchmark and four other multimodal understanding tasks
show significant improvements over baselines, proving that model composition
can create a versatile model capable of processing inputs from multiple
modalities.

在该篇论文中，研究人员介绍了一种新的范例通过现有 MLLMs 的模型组合来创建一个新模型，该模型保留了每个原始模型的模态理解能力。通过实验证明，模型组合可以创建一个多功能模型，能够处理来自多种模态的输入数据。

多模态大语言模型的模型组合

Model Composition for Multimodal Large Language Models

Diffusion models can be parameterised in terms of either a score or an energy
function. The energy parameterisation has better theoretical properties, mainly
that it enables an extended sampling procedure with a Metropolis--Hastings
correction step, based on the change in total energy in the proposed samples.
However, it seems to yield slightly worse performance, and more importantly,
due to the widespread popularity of score-based diffusion, there are limited
availability of off-the-shelf pre-trained energy-based ones. This limitation
undermines the purpose of model composition, which aims to combine pre-trained
models to sample from new distributions. Our proposal, however, suggests
retaining the score parameterization and instead computing the energy-based
acceptance probability through line integration of the score function. This
allows us to re-use existing diffusion models and still combine the reverse
process with various Markov-Chain Monte Carlo (MCMC) methods. We evaluate our
method on a 2D experiment and find that it achieve similar or arguably better
performance than the energy parameterisation.

通过对评分函数进行线积分计算来计算基于能量的接受概率，从而实现扩展的采样过程，使我们能够重复使用现有的扩散模型并将其与各种马尔科夫链蒙特卡洛方法结合。我们在一个二维实验中评估了我们的方法，并发现其达到了与能量参数化方法相似或更好的性能。

基于评分的扩散模型修正的 MCMC 方法用于模型组合

MCMC-Correction of Score-Based Diffusion Models for Model Composition

Artificial General Intelligence (AGI) requires comprehensive understanding
and generation capabilities for a variety of tasks spanning different
modalities and functionalities. Integrative AI is one important direction to
approach AGI, through combining multiple models to tackle complex multimodal
tasks. However, there is a lack of a flexible and composable platform to
facilitate efficient and effective model composition and coordination. In this
paper, we propose the i-Code Studio, a configurable and composable framework
for Integrative AI. The i-Code Studio orchestrates multiple pre-trained models
in a finetuning-free fashion to conduct complex multimodal tasks. Instead of
simple model composition, the i-Code Studio provides an integrative, flexible,
and composable setting for developers to quickly and easily compose
cutting-edge services and technologies tailored to their specific requirements.
The i-Code Studio achieves impressive results on a variety of zero-shot
multimodal tasks, such as video-to-text retrieval, speech-to-speech
translation, and visual question answering. We also demonstrate how to quickly
build a multimodal agent based on the i-Code Studio that can communicate and
personalize for users.

文章提出了一个配置和可组合的框架 i-Code Studio，用于处理 AGI 中的多模态任务，并且成功地在影片文本检索、语音翻译、视觉问答等多个任务中实现了出色的结果。