Large Language Models (LLMs), primarily trained on text-based datasets,
exhibit exceptional proficiencies in understanding and executing complex
linguistic instructions via text outputs. However, they falter when requests to
generate non-text ones. Concurrently, modality conversion models, such as
text-to-image, despite generating high-quality images, suffer from a lack of
extensive textual pretraining. As a result, these models are only capable of
accommodating specific image descriptions rather than comprehending more
complex instructions. To bridge this gap, we propose a novel approach,
\methodname, from a modality conversion perspective that evolves a text-based
LLM into a multi-modal one. We specifically employ a minimal dataset to
instruct LLMs to recognize the intended output modality as directed by the
instructions. Consequently, the adapted LLM can effectively summon various
off-the-shelf modality conversion models from the model zoos to generate
non-text responses. This circumvents the necessity for complicated pretraining
that typically requires immense quantities of paired multi-modal data, while
simultaneously inheriting the extensive knowledge of LLMs and the ability of
high-quality generative models. To evaluate and compare the adapted multi-modal
LLM with its traditional counterparts, we have constructed a multi-modal
instruction benchmark that solicits diverse modality outputs. The experiment
results reveal that, with minimal training, LLMs can be conveniently adapted to
comprehend requests for non-text responses, thus achieving higher flexibility
in multi-modal scenarios. Code and data will be made available at
this https URL

通过将大型语言模型转化为多模态模型，能够在无需复杂预训练的情况下，方便地为非文本响应提供高度灵活性的多模态方案。