Video-to-video synthesis models face significant challenges, such as ensuring
consistent character generation across frames, maintaining smooth temporal
transitions, and preserving quality during fast motion. The introduction of
joint fully cross-frame self-attention mechanisms has improved character
consistency, but this comes at the cost of increased computational complexity.
This full cross-frame self-attention mechanism also incorporates redundant
details and limits the number of frames that can be jointly edited due to its
computational cost. Moreover, the lack of frames in cross-frame attention
adversely affects temporal consistency and visual quality. To address these
limitations, we propose a new adaptive motion-guided cross-frame attention
mechanism that drastically reduces complexity while preserving semantic details
and temporal consistency. Specifically, we selectively incorporate the moving
regions of successive frames in cross-frame attention and sparsely include
stationary regions based on optical flow sampling. This technique allows for an
increased number of jointly edited frames without additional computational
overhead. For longer duration of video editing, existing methods primarily
focus on frame interpolation or flow-warping from jointly edited keyframes,
which often results in blurry frames or reduced temporal consistency. To
improve this, we introduce KV-caching of jointly edited frames and reuse the
same KV across all intermediate frames, significantly enhancing both
intermediate frame quality and temporal consistency. Overall, our
motion-sampling method enables the use of around three times more keyframes
than existing joint editing methods while maintaining superior prediction
quality. Ada-VE achieves up to 4x speed-up when using fully-extended
self-attention across 40 frames for joint editing, without compromising visual
quality or temporal consistency.

通过引入自适应的运动引导的跨帧注意机制，本文提出了一种显著降低计算复杂性的方法，同时保留语义细节和时间一致性，并通过 KV 缓存共享提高了中间帧质量和时间一致性，大大增加了联合编辑的关键帧数量，不受视觉质量和时间一致性的影响。