Video-to-video synthesis models face significant challenges, such as ensuring
consistent character generation across frames, maintaining smooth temporal
transitions, and preserving quality during fast motion. The introduction of
joint fully cross-frame self-attention mechanisms has improved character
consistency, but this comes at the cost of increased computational complexity.
This full cross-frame self-attention mechanism also incorporates redundant
details and limits the number of frames that can be jointly edited due to its
computational cost. Moreover, the lack of frames in cross-frame attention
adversely affects temporal consistency and visual quality. To address these
limitations, we propose a new adaptive motion-guided cross-frame attention
mechanism that drastically reduces complexity while preserving semantic details
and temporal consistency. Specifically, we selectively incorporate the moving
regions of successive frames in cross-frame attention and sparsely include
stationary regions based on optical flow sampling. This technique allows for an
increased number of jointly edited frames without additional computational
overhead. For longer duration of video editing, existing methods primarily
focus on frame interpolation or flow-warping from jointly edited keyframes,
which often results in blurry frames or reduced temporal consistency. To
improve this, we introduce KV-caching of jointly edited frames and reuse the
same KV across all intermediate frames, significantly enhancing both
intermediate frame quality and temporal consistency. Overall, our
motion-sampling method enables the use of around three times more keyframes
than existing joint editing methods while maintaining superior prediction
quality. Ada-VE achieves up to 4x speed-up when using fully-extended
self-attention across 40 frames for joint editing, without compromising visual
quality or temporal consistency.

通过引入自适应的运动引导的跨帧注意机制，本文提出了一种显著降低计算复杂性的方法，同时保留语义细节和时间一致性，并通过 KV 缓存共享提高了中间帧质量和时间一致性，大大增加了联合编辑的关键帧数量，不受视觉质量和时间一致性的影响。

Ada-VE: 自适应运动先验的无需训练的一致性视频编辑

Ada-VE: Training-Free Consistent Video Editing Using Adaptive Motion  Prior

Leveraging the generative ability of image diffusion models offers great
potential for zero-shot video-to-video translation. The key lies in how to
maintain temporal consistency across generated video frames by image diffusion
models. Previous methods typically adopt cross-frame attention, \emph{i.e.,}
sharing the \textit{key} and \textit{value} tokens across attentions of
different frames, to encourage the temporal consistency. However, in those
works, temporal inconsistency issue may not be thoroughly solved, rendering the
fidelity of generated videos limited.%The current state of the art cross-frame
attention method aims at maintaining fine-grained visual details across frames,
but it is still challenged by the temporal coherence problem. In this paper, we
find the bottleneck lies in the unconstrained query tokens and propose a new
zero-shot video-to-video translation framework, named \textit{LatentWarp}. Our
approach is simple: to constrain the query tokens to be temporally consistent,
we further incorporate a warping operation in the latent space to constrain the
query tokens. Specifically, based on the optical flow obtained from the
original video, we warp the generated latent features of last frame to align
with the current frame during the denoising process. As a result, the
corresponding regions across the adjacent frames can share closely-related
query tokens and attention outputs, which can further improve latent-level
consistency to enhance visual temporal coherence of generated videos. Extensive
experiment results demonstrate the superiority of \textit{LatentWarp} in
achieving video-to-video translation with temporal coherence.

通过图像扩散模型的生成能力，我们提出了一种名为 LatentWarp 的新的零样本视频 - 视频翻译框架，通过约束查询令牌的时间一致性，在潜在空间中进一步结合了变形操作以约束查询令牌，从而实现了生成视频的视觉时间相干性的提升。

潜在变形：用于零样本视频到视频翻译的一致性扩散潜变量

LatentWarp: Consistent Diffusion Latents for Zero-Shot Video-to-Video  Translation

Optical flow estimation aims to find the 2D motion field by identifying
corresponding pixels between two images. Despite the tremendous progress of
deep learning-based optical flow methods, it remains a challenge to accurately
estimate large displacements with motion blur. This is mainly because the
correlation volume, the basis of pixel matching, is computed as the dot product
of the convolutional features of the two images. The locality of convolutional
features makes the computed correlations susceptible to various noises. On
large displacements with motion blur, noisy correlations could cause severe
errors in the estimated flow. To overcome this challenge, we propose a new
architecture "CRoss-Attentional Flow Transformer" (CRAFT), aiming to revitalize
the correlation volume computation. In CRAFT, a Semantic Smoothing Transformer
layer transforms the features of one frame, making them more global and
semantically stable. In addition, the dot-product correlations are replaced
with transformer Cross-Frame Attention. This layer filters out feature noises
through the Query and Key projections, and computes more accurate correlations.
On Sintel (Final) and KITTI (foreground) benchmarks, CRAFT has achieved new
state-of-the-art performance. Moreover, to test the robustness of different
models on large motions, we designed an image shifting attack that shifts input
images to generate large artificial motions. Under this attack, CRAFT performs
much more robustly than two representative methods, RAFT and GMA. The code of
CRAFT is is available at this https URL

提出了一种新型的 CRoss-Attentional Flow Transformer (CRAFT) 系统，该系统利用语义平滑器和变形器跨框架注意力代替点积相关性来估计光流。与其他深度学习光流方法相比，CRAFT 在对大运动进行准确估计时效果更佳，并在 Sintel（Final）和 KITTI（foreground）基准测试中创造了新的最优表现。