Text-to-image (T2I) generation models have significantly advanced in recent
years. However, effective interaction with these models is challenging for
average users due to the need for specialized prompt engineering knowledge and
the inability to perform multi-turn image generation, hindering a dynamic and
iterative creation process. Recent attempts have tried to equip Multi-modal
Large Language Models (MLLMs) with T2I models to bring the user's natural
language instructions into reality. Hence, the output modality of MLLMs is
extended, and the multi-turn generation quality of T2I models is enhanced
thanks to the strong multi-modal comprehension ability of MLLMs. However, many
of these works face challenges in identifying correct output modalities and
generating coherent images accordingly as the number of output modalities
increases and the conversations go deeper. Therefore, we propose DialogGen, an
effective pipeline to align off-the-shelf MLLMs and T2I models to build a
Multi-modal Interactive Dialogue System (MIDS) for multi-turn Text-to-Image
generation. It is composed of drawing prompt alignment, careful training data
curation, and error correction. Moreover, as the field of MIDS flourishes,
comprehensive benchmarks are urgently needed to evaluate MIDS fairly in terms
of output modality correctness and multi-modal output coherence. To address
this issue, we introduce the Multi-modal Dialogue Benchmark (DialogBen), a
comprehensive bilingual benchmark designed to assess the ability of MLLMs to
generate accurate and coherent multi-modal content that supports image editing.
It contains two evaluation metrics to measure the model's ability to switch
modalities and the coherence of the output images. Our extensive experiments on
DialogBen and user study demonstrate the effectiveness of DialogGen compared
with other State-of-the-Art models.

通过将多模态大型语言模型与文本到图像生成模型相结合，提出了一种用于多轮文本到图像生成的多模态交互对话系统，同时引入了一种全面的多模态对话评估基准，以评估模型在生成准确且连贯的多模态内容上的能力，包括模态切换和输出图像的连贯性。