Current state-of-the-art methods for video inpainting typically rely on
optical flow or attention-based approaches to inpaint masked regions by
propagating visual information across frames. While such approaches have led to
significant progress on standard benchmarks, they struggle with tasks that
require the synthesis of novel content that is not present in other frames. In
this paper we reframe video inpainting as a conditional generative modeling
problem and present a framework for solving such problems with conditional
video diffusion models. We highlight the advantages of using a generative
approach for this task, showing that our method is capable of generating
diverse, high-quality inpaintings and synthesizing new content that is
spatially, temporally, and semantically consistent with the provided context.

通过将视频修复问题作为条件生成建模问题加以解决的框架，以及利用生成方法的优势，本文展示了能够生成多样化且高质量修复效果的方法，并能够在时间、空间和语义上与给定的上下文相协调地合成新内容。

基于条件扩散模型的语义一致视频修复

Semantically Consistent Video Inpainting with Conditional Diffusion  Models

Many video editing tasks such as rotoscoping or object removal require the
propagation of context across frames. While transformers and other
attention-based approaches that aggregate features globally have demonstrated
great success at propagating object masks from keyframes to the whole video,
they struggle to propagate high-frequency details such as textures faithfully.
We hypothesize that this is due to an inherent bias of global attention towards
low-frequency features. To overcome this limitation, we present a two-stream
approach, where high-frequency features interact locally and low-frequency
features interact globally. The global interaction stream remains robust in
difficult situations such as large camera motions, where explicit alignment
fails. The local interaction stream propagates high-frequency details through
deformable feature aggregation and, informed by the global interaction stream,
learns to detect and correct errors of the deformation field. We evaluate our
two-stream approach for inpainting tasks, where experiments show that it
improves both the propagation of features within a single frame as required for
image inpainting, as well as their propagation from keyframes to target frames.
Applied to video inpainting, our approach leads to 44% and 26% improvements in
FID and LPIPS scores. Code at this https URL

该研究提出了一种双流方法，其中高频特征实现局部交互，低频特征实现全局交互，旨在解决全局注意力对于细节的无法恰当处理的困境，并在修补任务中对其进行了评估