Video editing stands as a cornerstone of digital media, from entertainment
and education to professional communication. However, previous methods often
overlook the necessity of comprehensively understanding both global and local
contexts, leading to inaccurate and inconsistency edits in the spatiotemporal
dimension, especially for long videos. In this paper, we introduce VIA, a
unified spatiotemporal VIdeo Adaptation framework for global and local video
editing, pushing the limits of consistently editing minute-long videos. First,
to ensure local consistency within individual frames, the foundation of VIA is
a novel test-time editing adaptation method, which adapts a pre-trained image
editing model for improving consistency between potential editing directions
and the text instruction, and adapts masked latent variables for precise local
control. Furthermore, to maintain global consistency over the video sequence,
we introduce spatiotemporal adaptation that adapts consistent attention
variables in key frames and strategically applies them across the whole
sequence to realize the editing effects. Extensive experiments demonstrate
that, compared to baseline methods, our VIA approach produces edits that are
more faithful to the source videos, more coherent in the spatiotemporal
context, and more precise in local control. More importantly, we show that VIA
can achieve consistent long video editing in minutes, unlocking the potentials
for advanced video editing tasks over long video sequences.

VIA 是一个统一的时空 VIdeo 适应框架，其基础是一种新的测试时间编辑适应方法，用于通过适应预训练的图像编辑模型来提高潜在编辑方向和文本指令之间的一致性，实现精确的本地控制，同时通过进行一致的关注变量适应来维护整个视频序列的全局一致性，以实现编辑效果。实验结果表明，与基准方法相比，使用 VIA 方法进行的编辑更加忠实于源视频，在时空上更加连贯，并具有更精确的本地控制。更重要的是，我们展示了 VIA 可以在几分钟内实现一致的长视频编辑，释放了对长视频序列的高级视频编辑任务的潜力。

VIA：全球和本地视频编辑的时空视频调适框架

VIA: A Spatiotemporal Video Adaptation Framework for Global and Local  Video Editing

Leveraging pre-trained conditional diffusion models for video editing without
further tuning has gained increasing attention due to its promise in film
production, advertising, etc. Yet, seminal works in this line fall short in
generation length, temporal coherence, or fidelity to the source video. This
paper aims to bridge the gap, establishing a simple and effective baseline for
training-free diffusion model-based long video editing. As suggested by prior
arts, we build the pipeline upon ControlNet, which excels at various image
editing tasks based on text prompts. To break down the length constraints
caused by limited computational memory, we split the long video into
consecutive windows and develop a novel cross-window attention mechanism to
ensure the consistency of global style and maximize the smoothness among
windows. To achieve more accurate control, we extract the information from the
source video via DDIM inversion and integrate the outcomes into the latent
states of the generations. We also incorporate a video frame interpolation
model to mitigate the frame-level flickering issue. Extensive empirical studies
verify the superior efficacy of our method over competing baselines across
scenarios, including the replacement of the attributes of foreground objects,
style transfer, and background replacement. In particular, our method manages
to edit videos with up to 128 frames according to user requirements. Code is
available at this https URL

本研究旨在建立一个简单而有效的基于训练无关扩散模型的长视频编辑基线，通过分割长视频并采用新颖的跨窗口注意机制，以确保全局风格的一致性和窗口之间的平滑性。通过 DDIM 反演从源视频中提取信息，并将结果整合到生成的潜在状态中，同时结合视频帧插值模型以减轻帧级闪烁问题。经过广泛的实证研究，验证了我们的方法在替换前景物体属性、风格转移和背景替换等场景下相对竞争基准的卓越有效性。