Leveraging the generative ability of image diffusion models offers great
potential for zero-shot video-to-video translation. The key lies in how to
maintain temporal consistency across generated video frames by image diffusion
models. Previous methods typically adopt cross-frame attention, \emph{i.e.,}
sharing the \textit{key} and \textit{value} tokens across attentions of
different frames, to encourage the temporal consistency. However, in those
works, temporal inconsistency issue may not be thoroughly solved, rendering the
fidelity of generated videos limited.%The current state of the art cross-frame
attention method aims at maintaining fine-grained visual details across frames,
but it is still challenged by the temporal coherence problem. In this paper, we
find the bottleneck lies in the unconstrained query tokens and propose a new
zero-shot video-to-video translation framework, named \textit{LatentWarp}. Our
approach is simple: to constrain the query tokens to be temporally consistent,
we further incorporate a warping operation in the latent space to constrain the
query tokens. Specifically, based on the optical flow obtained from the
original video, we warp the generated latent features of last frame to align
with the current frame during the denoising process. As a result, the
corresponding regions across the adjacent frames can share closely-related
query tokens and attention outputs, which can further improve latent-level
consistency to enhance visual temporal coherence of generated videos. Extensive
experiment results demonstrate the superiority of \textit{LatentWarp} in
achieving video-to-video translation with temporal coherence.

通过图像扩散模型的生成能力，我们提出了一种名为 LatentWarp 的新的零样本视频 - 视频翻译框架，通过约束查询令牌的时间一致性，在潜在空间中进一步结合了变形操作以约束查询令牌，从而实现了生成视频的视觉时间相干性的提升。

潜在变形：用于零样本视频到视频翻译的一致性扩散潜变量

LatentWarp: Consistent Diffusion Latents for Zero-Shot Video-to-Video  Translation

Large text-to-image diffusion models have exhibited impressive proficiency in
generating high-quality images. However, when applying these models to video
domain, ensuring temporal consistency across video frames remains a formidable
challenge. This paper proposes a novel zero-shot text-guided video-to-video
translation framework to adapt image models to videos. The framework includes
two parts: key frame translation and full video translation. The first part
uses an adapted diffusion model to generate key frames, with hierarchical
cross-frame constraints applied to enforce coherence in shapes, textures and
colors. The second part propagates the key frames to other frames with
temporal-aware patch matching and frame blending. Our framework achieves global
style and local texture temporal consistency at a low cost (without re-training
or optimization). The adaptation is compatible with existing image diffusion
techniques, allowing our framework to take advantage of them, such as
customizing a specific subject with LoRA, and introducing extra spatial
guidance with ControlNet. Extensive experimental results demonstrate the
effectiveness of our proposed framework over existing methods in rendering
high-quality and temporally-coherent videos.

本研究提出了一种基于零样本文本引导的视频到视频转换框架来适应图像模型到视频的应用，在形状、纹理和颜色上强制执行分层的跨帧约束，实现低成本下的全局样式和局部纹理的时空一致性。