As the capabilities of Large-Language Models (LLMs) become widely recognized,
there is an increasing demand for human-machine chat applications. Human
interaction with text often inherently invokes mental imagery, an aspect that
existing LLM-based chatbots like GPT-4 do not currently emulate, as they are
confined to generating text-only content. To bridge this gap, we introduce
ChatIllusion, an advanced Generative multimodal large language model (MLLM)
that combines the capabilities of LLM with not only visual comprehension but
also creativity. Specifically, ChatIllusion integrates Stable Diffusion XL and
Llama, which have been fine-tuned on modest image-caption data, to facilitate
multiple rounds of illustrated chats. The central component of ChatIllusion is
the "GenAdapter," an efficient approach that equips the multimodal language
model with capabilities for visual representation, without necessitating
modifications to the foundational model. Extensive experiments validate the
efficacy of our approach, showcasing its ability to produce diverse and
superior-quality image outputs Simultaneously, it preserves semantic
consistency and control over the dialogue, significantly enhancing the overall
user's quality of experience (QoE). The code is available at
this https URL

通过 GenAdapter 将视觉表示能力集成到 ChatIllusion 中，ChatIllusion 是一种高级生成多模态大语言模型，结合了大语言模型的能力和视觉理解能力，能够为用户提供多样且高质量的图像输出，同时保持语义一致性和对话控制性，极大提升用户的体验质量。