Leveraging the generative ability of image diffusion models offers great
potential for zero-shot video-to-video translation. The key lies in how to
maintain temporal consistency across generated video frames by image diffusion
models. Previous methods typically adopt cross-frame attention, \emph{i.e.,}
sharing the \textit{key} and \textit{value} tokens across attentions of
different frames, to encourage the temporal consistency. However, in those
works, temporal inconsistency issue may not be thoroughly solved, rendering the
fidelity of generated videos limited.%The current state of the art cross-frame
attention method aims at maintaining fine-grained visual details across frames,
but it is still challenged by the temporal coherence problem. In this paper, we
find the bottleneck lies in the unconstrained query tokens and propose a new
zero-shot video-to-video translation framework, named \textit{LatentWarp}. Our
approach is simple: to constrain the query tokens to be temporally consistent,
we further incorporate a warping operation in the latent space to constrain the
query tokens. Specifically, based on the optical flow obtained from the
original video, we warp the generated latent features of last frame to align
with the current frame during the denoising process. As a result, the
corresponding regions across the adjacent frames can share closely-related
query tokens and attention outputs, which can further improve latent-level
consistency to enhance visual temporal coherence of generated videos. Extensive
experiment results demonstrate the superiority of \textit{LatentWarp} in
achieving video-to-video translation with temporal coherence.

通过图像扩散模型的生成能力，我们提出了一种名为 LatentWarp 的新的零样本视频 - 视频翻译框架，通过约束查询令牌的时间一致性，在潜在空间中进一步结合了变形操作以约束查询令牌，从而实现了生成视频的视觉时间相干性的提升。