Large language models (LLMs) have demonstrated impressive zero-shot abilities
on a variety of open-ended tasks, while recent research has also explored the
use of LLMs for multi-modal generation. In this study, we introduce mPLUG-Owl,
a novel training paradigm that equips LLMs with multi-modal abilities through
modularized learning of foundation LLM, a visual knowledge module, and a visual
abstractor module. This approach can support multiple modalities and facilitate
diverse unimodal and multimodal abilities through modality collaboration. The
training paradigm of mPLUG-Owl involves a two-stage method for aligning image
and text, which learns visual knowledge with the assistance of LLM while
maintaining and even improving the generation abilities of LLM. In the first
stage, the visual knowledge module and abstractor module are trained with a
frozen LLM module to align the image and text. In the second stage,
language-only and multi-modal supervised datasets are used to jointly fine-tune
a low-rank adaption (LoRA) module on LLM and the abstractor module by freezing
the visual knowledge module. We carefully build a visually-related instruction
evaluation set OwlEval. Experimental results show that our model outperforms
existing multi-modal models, demonstrating mPLUG-Owl's impressive instruction
and visual understanding ability, multi-turn conversation ability, and
knowledge reasoning ability. Besides, we observe some unexpected and exciting
abilities such as multi-image correlation and scene text understanding, which
makes it possible to leverage it for harder real scenarios, such as vision-only
document comprehension. Our code, pre-trained model, instruction-tuned models,
and evaluation set are available at this https URL The
online demo is available at this https URL

本研究介绍了一种新的培训范式 mPLUG-Owl，该方法通过基于模块化的学习将 Large language models（LLMs）配备了多模态能力。实验表明，通过此方法可以获得多种单模态和多模态能力，包括指导视觉能力、多回合对话和知识推理等，同时出现了一些意想不到且令人兴奋的特性，如多图像相关性和场景文本理解。