This work investigates a challenging task named open-domain interleaved
image-text generation, which generates interleaved texts and images following
an input query. We propose a new interleaved generation framework based on
prompting large-language models (LLMs) and pre-trained text-to-image (T2I)
models, namely OpenLEAF. In OpenLEAF, the LLM generates textual descriptions,
coordinates T2I models, creates visual prompts for generating images, and
incorporates global contexts into the T2I models. This global context improves
the entity and style consistencies of images in the interleaved generation. For
model assessment, we first propose to use large multi-modal models (LMMs) to
evaluate the entity and style consistencies of open-domain interleaved
image-text sequences. According to the LMM evaluation on our constructed
evaluation set, the proposed interleaved generation framework can generate
high-quality image-text content for various domains and applications, such as
how-to question answering, storytelling, graphical story rewriting, and
webpage/poster generation tasks. Moreover, we validate the effectiveness of the
proposed LMM evaluation technique with human assessment. We hope our proposed
framework, benchmark, and LMM evaluation could help establish the intriguing
interleaved image-text generation task.

这项工作研究了一项名为开放域交错图像 - 文本生成的挑战性任务，该任务根据输入查询生成交错的文本和图像。我们提出了一种新的基于大型语言模型（LLMs）和预训练的文本到图像模型（T2I）的交错生成框架，即 OpenLEAF。在 OpenLEAF 中，LLM 生成文本描述，协调 T2I 模型，创建用于生成图像的视觉提示，并将全局上下文融入到 T2I 模型中。这种全局上下文提高了交错生成中图像的实体和风格的一致性。我们首先提出使用大型多模态模型（LMMs）来评估开放域交错图像 - 文本序列的实体和风格的一致性。根据 LMM 在我们构建的评估集上的评估，所提出的交错生成框架可以为各种领域和应用生成高质量的图像 - 文本内容，例如如何问答、叙事、图形故事改写以及网页 / 海报生成任务。此外，我们通过人类评估验证了所提出的 LMM 评估技术的有效性。我们希望我们提出的框架、基准和 LMM 评估能够帮助建立有趣的交错图像 - 文本生成任务。