In this work, we study the problem of generating novel images from complex
multimodal prompt sequences. While existing methods achieve promising results
for text-to-image generation, they often struggle to capture fine-grained
details from lengthy prompts and maintain contextual coherence within prompt
sequences. Moreover, they often result in misaligned image generation for
prompt sequences featuring multiple objects. To address this, we propose a
Multi-modal Generation via Cross-Modal In-Context Learning (MGCC) method that
generates novel images from complex multimodal prompt sequences by leveraging
the combined capabilities of large language models (LLMs) and diffusion models.
Our MGCC comprises a novel Cross-Modal Refinement module to explicitly learn
cross-modal dependencies between the text and image in the LLM embedding space,
and a contextual object grounding module to generate object bounding boxes
specifically targeting scenes with multiple objects. Our MGCC demonstrates a
diverse range of multimodal capabilities, like novel image generation, the
facilitation of multimodal dialogue, and generation of texts. Experimental
evaluations on two benchmark datasets, demonstrate the effectiveness of our
method. On Visual Story Generation (VIST) dataset with multimodal inputs, our
MGCC achieves a CLIP Similarity score of $0.652$ compared to SOTA GILL $0.641$.
Similarly, on Visual Dialogue Context (VisDial) having lengthy dialogue
sequences, our MGCC achieves an impressive CLIP score of $0.660$, largely
outperforming existing SOTA method scoring $0.645$. Code:
this https URL

本研究提出了一种利用大型语言模型和扩散模型的多模式生成方法（MGCC），通过在 LLM 嵌入空间中显式学习文本和图像之间的跨模式依赖关系以及生成特定于多物体场景的对象边界框，实现了从复杂的多模式提示序列中生成新图像的能力，并在两个基准数据集上进行了实验验证。

跨模态上下文学习实现多模态生成

Multi-modal Generation via Cross-Modal In-Context Learning

In recent years, latent variable models, such as the Conditional Variational
Auto Encoder (CVAE), have been applied to both personalized and empathetic
dialogue generation. Prior work have largely focused on generating diverse
dialogue responses that exhibit persona consistency and empathy. However, when
it comes to the contextual coherence of the generated responses, there is still
room for improvement. Hence, to improve the contextual coherence, we propose a
novel Uncertainty Aware CVAE (UA-CVAE) framework. The UA-CVAE framework
involves approximating and incorporating the aleatoric uncertainty during
response generation. We apply our framework to both personalized and empathetic
dialogue generation. Empirical results show that our framework significantly
improves the contextual coherence of the generated response. Additionally, we
introduce a novel automatic metric for measuring contextual coherence, which
was found to correlate positively with human judgement.

通过引入 Uncertainty Aware CVAE 框架来近似和整合响应生成过程中的 aleatoric 不确定性，改善个性化和移情式对话生成中生成响应的上下文连贯性。同时，研究者们提出了一种新的自动度量方法来衡量生成响应的上下文连贯性，并且该度量与人类的判断显著相关。

提升变分个性化和移情对话代理的上下文连贯性

Improving Contextual Coherence in Variational Personalized and Empathetic Dialogue Agents

In multi-modal dialogue systems, it is important to allow the use of images
as part of a multi-turn conversation. Training such dialogue systems generally
requires a large-scale dataset consisting of multi-turn dialogues that involve
images, but such datasets rarely exist. In response, this paper proposes a 45k
multi-modal dialogue dataset created with minimal human intervention. Our
method to create such a dataset consists of (1) preparing and pre-processing
text dialogue datasets, (2) creating image-mixed dialogues by using a
text-to-image replacement technique, and (3) employing a
contextual-similarity-based filtering step to ensure the contextual coherence
of the dataset. To evaluate the validity of our dataset, we devise a simple
retrieval model for dialogue sentence prediction tasks. Automatic metrics and
human evaluation results on such tasks show that our dataset can be effectively
used as training data for multi-modal dialogue systems which require an
understanding of images and text in a context-aware manner. Our dataset and
generation code is available at
this https URL

本文提出创建一种包含图像的 45k 多模态对话数据集的方法，该方法通过准备、预处理文本对话数据集、使用文本转图像技术创建混合图像对话，以及利用基于上下文相似性的过滤步骤确保数据集的上下文一致性。自动度量和人类评估结果表明，我们的数据集可以有效地用于多模态对话系统的训练，需要以上下文感知的方式理解图像和文本。