Diffusion models have revolutionized text-driven video editing. However,
applying these methods to real-world editing encounters two significant
challenges: (1) the rapid increase in graphics memory demand as the number of
frames grows, and (2) the inter-frame inconsistency in edited videos. To this
end, we propose NVEdit, a novel text-driven video editing framework designed to
mitigate memory overhead and improve consistent editing for real-world long
videos. Specifically, we construct a neural video field, powered by tri-plane
and sparse grid, to enable encoding long videos with hundreds of frames in a
memory-efficient manner. Next, we update the video field through off-the-shelf
Text-to-Image (T2I) models to impart text-driven editing effects. A progressive
optimization strategy is developed to preserve original temporal priors.
Importantly, both the neural video field and T2I model are adaptable and
replaceable, thus inspiring future research. Experiments demonstrate that our
approach successfully edits hundreds of frames with impressive inter-frame
consistency.

本文介绍了一种名为 NVEdit 的新型文本驱动视频编辑框架，通过构建神经视频场、采用渐进优化策略以及替换可适应的神经视频场和文本到图像模型，解决了实际场景中长视频编辑中的图形内存需求快速增长和编辑的帧间不一致性等挑战，实现了对具有显著帧间一致性的长视频的成功编辑。

神经视频场编辑

Neural Video Fields Editing

The generative AI revolution has recently expanded to videos. Nevertheless,
current state-of-the-art video models are still lagging behind image models in
terms of visual quality and user control over the generated content. In this
work, we present a framework that harnesses the power of a text-to-image
diffusion model for the task of text-driven video editing. Specifically, given
a source video and a target text-prompt, our method generates a high-quality
video that adheres to the target text, while preserving the spatial layout and
motion of the input video. Our method is based on a key observation that
consistency in the edited video can be obtained by enforcing consistency in the
diffusion feature space. We achieve this by explicitly propagating diffusion
features based on inter-frame correspondences, readily available in the model.
Thus, our framework does not require any training or fine-tuning, and can work
in conjunction with any off-the-shelf text-to-image editing method. We
demonstrate state-of-the-art editing results on a variety of real-world videos.
Webpage: this https URL

基于文本驱动的视频编辑，我们介绍了一个利用文本到图像扩散模型的框架，生成高质量视频的同时保留输入视频的空间布局和运动，实现编辑视频的一致性。