Building general-purpose models that can perceive diverse real-world
modalities and solve various tasks is an appealing target in artificial
intelligence. In this paper, we present ChatBridge, a novel multimodal language
model that leverages the expressive capabilities of language as the catalyst to
bridge the gap between various modalities. We show that only language-paired
two-modality data is sufficient to connect all modalities. ChatBridge leverages
recent large language models (LLM) and extends their zero-shot capabilities to
incorporate diverse multimodal inputs. ChatBridge undergoes a two-stage
training. The first stage aligns each modality with language, which brings
emergent multimodal correlation and collaboration abilities. The second stage
instruction-finetunes ChatBridge to align it with user intent with our newly
proposed multimodal instruction tuning dataset, named MULTIS, which covers a
wide range of 16 multimodal tasks of text, image, video, and audio modalities.
We show strong quantitative and qualitative results on zero-shot multimodal
tasks covering text, image, video, and audio modalities. All codes, data, and
models of ChatBridge will be open-sourced.

本文提出了一种名为 ChatBridge 的多模态语言模型，利用语言的表达能力作为各种模态之间的桥梁，从而实现多种任务的感知。ChatBridge 利用大型语言模型（LLM）的零 - shot 能力来处理多样化的多模态输入，尤其是在涵盖文本、图像、视频和音频模态的 16 种多模态任务中展现出良好的强化学习效果。