Although instruction-tuned large language models (LLMs) have exhibited
remarkable capabilities across various NLP tasks, their effectiveness on other
data modalities beyond text has not been fully studied. In this work, we
propose Macaw-LLM, a novel multi-modal LLM that seamlessly integrates visual,
audio, and textual information. Macaw-LLM consists of three main components: a
modality module for encoding multi-modal data, a cognitive module for
harnessing pretrained LLMs, and an alignment module for harmonizing diverse
representations. Our novel alignment module seamlessly bridges multi-modal
features to textual features, simplifying the adaptation process from the
modality modules to the cognitive module. In addition, we construct a
large-scale multi-modal instruction dataset in terms of multi-turn dialogue,
including 69K image instances and 50K video instances. We have made our data,
code and model publicly available, which we hope can pave the way for future
research in multi-modal LLMs and expand the capabilities of LLMs to handle
diverse data modalities and address complex real-world scenarios.

本研究提出 Macaw-LLM，一种新型的多模式学习模型，可以无缝整合视觉、音频和文本信息，其中主要包括三个模块：一种用于编码多模式数据的模态模块，一种利用预训练 LLM 的认知模块，和一种用于协调多样化表示形式的对齐模块，并在此基础上构建了一个大规模的多模式指令数据集以应对复杂的实际场景。