Large text-to-image diffusion models have achieved remarkable success in
generating diverse high-quality images that are closely aligned with text
prompt. But, when these models applied to video the main challenge is to ensure
temporal consistency and coherent editing. In this paper, we proposed InFusion,
a framework for zero-shot text-based video editing leveraging large pre-trained
image diffusion models. Our framework specifically supports editing of multiple
concepts with the pixel level control over diverse concepts mentioned in the
editing prompt. Specifically, we inject the difference of features from U-Net
residual blocks in decoder layers for source and edit prompt, this when
combined with injected attention features make it feasible to query the source
contents and scale the edited concepts along with injection of unedited parts.
The editing is further controlled in fine-grained manner with mask extraction
and attention fusion strategy which cuts the edited part from source and paste
it from the denoising pipeline for target prompt. Our framework is a low cost
alternative for the one-shot tuned models for editing since it does not require
training. We demonstrated the complex concept editing with generalised image
model (Stable Diffusion v1.5) using LoRA. Adaptation is compatible with all the
existing image diffusion techniques. Extensive experimental results demonstrate
the effectiveness over existing methods in rendering high-quality and
temporally consistent videos.

使用 InFusion 框架，基于大型预训练图像扩散模型进行零样例基于文本的视频编辑，以实现丰富的概念编辑、时间一致性和连贯性。