Conditional diffusion models have exhibited superior performance in
high-fidelity text-guided visual generation and editing. Nevertheless,
prevailing text-guided visual diffusion models primarily focus on incorporating
text-visual relationships exclusively into the reverse process, often
disregarding their relevance in the forward process. This inconsistency between
forward and reverse processes may limit the precise conveyance of textual
semantics in visual synthesis results. To address this issue, we propose a
novel and general contextualized diffusion model (ContextDiff) by incorporating
the cross-modal context encompassing interactions and alignments between text
condition and visual sample into forward and reverse processes. We propagate
this context to all timesteps in the two processes to adapt their trajectories,
thereby facilitating cross-modal conditional modeling. We generalize our
contextualized diffusion to both DDPMs and DDIMs with theoretical derivations,
and demonstrate the effectiveness of our model in evaluations with two
challenging tasks: text-to-image generation, and text-to-video editing. In each
task, our ContextDiff achieves new state-of-the-art performance, significantly
enhancing the semantic alignment between text condition and generated samples,
as evidenced by quantitative and qualitative evaluations. Our code is available
at this https URL

我们提出了一种新颖而通用的上下文扩散模型 (ContextDiff)，通过将文本条件与视觉样本之间的交互和对齐引入前向和后向过程中，将上下文传播到这两个过程的所有时间步，从而促进跨模态条件建模，在文本到图像生成和文本到视频编辑的两个挑战性任务中，我们的 ContextDiff 实现了新的最先进性能，通过定量和定性评估明显增强了文本条件与生成样本之间的语义对齐。

跨模态语境扩散模型的文本导向视觉生成与编辑

Cross-Modal Contextualized Diffusion Models for Text-Guided Visual  Generation and Editing

Text-to-video editing aims to edit the visual appearance of a source video
conditional on textual prompts. A major challenge in this task is to ensure
that all frames in the edited video are visually consistent. Most recent works
apply advanced text-to-image diffusion models to this task by inflating 2D
spatial attention in the U-Net into spatio-temporal attention. Although
temporal context can be added through spatio-temporal attention, it may
introduce some irrelevant information for each patch and therefore cause
inconsistency in the edited video. In this paper, for the first time, we
introduce optical flow into the attention module in the diffusion model's U-Net
to address the inconsistency issue for text-to-video editing. Our method,
FLATTEN, enforces the patches on the same flow path across different frames to
attend to each other in the attention module, thus improving the visual
consistency in the edited videos. Additionally, our method is training-free and
can be seamlessly integrated into any diffusion-based text-to-video editing
methods and improve their visual consistency. Experiment results on existing
text-to-video editing benchmarks show that our proposed method achieves the new
state-of-the-art performance. In particular, our method excels in maintaining
the visual consistency in the edited videos.

FLATTEN 是一种用于文本到视频编辑的训练无关方法，通过在扩散模型的 U-Net 中引入光流来提高编辑视频的视觉一致性，并且能够无缝集成到任何基于扩散的文本到视频编辑方法中，提高其视觉一致性。