Video composition is the core task of video editing. Although image
composition based on diffusion models has been highly successful, it is not
straightforward to extend the achievement to video object composition tasks,
which not only exhibit corresponding interaction effects but also ensure that
the objects in the composited video maintain motion and identity consistency,
which is necessary to composite a physical harmony video. To address this
challenge, we propose a Multiple Video Object Composition (MVOC) method based
on diffusion models. Specifically, we first perform DDIM inversion on each
video object to obtain the corresponding noise features. Secondly, we combine
and edit each object by image editing methods to obtain the first frame of the
composited video. Finally, we use the image-to-video generation model to
composite the video with feature and attention injections in the Video Object
Dependence Module, which is a training-free conditional guidance operation for
video generation, and enables the coordination of features and attention maps
between various objects that can be non-independent in the composited video.
The final generative model not only constrains the objects in the generated
video to be consistent with the original object motion and identity, but also
introduces interaction effects between objects. Extensive experiments have
demonstrated that the proposed method outperforms existing state-of-the-art
approaches. Project page: this https URL

基于扩散模型的视频多目标合成方法通过图像编辑和视频生成模型在保持物体运动和身份连贯性的同时引入物体之间的相互作用效果，从而实现了视频合成的物理和谐。

MVOC: 一种无需训练的多视频对象组合方法与扩散模型

MVOC: a training-free multiple video object composition method with  diffusion models

The video composition task aims to integrate specified foregrounds and
backgrounds from different videos into a harmonious composite. Current
approaches, predominantly trained on videos with adjusted foreground color and
lighting, struggle to address deep semantic disparities beyond superficial
adjustments, such as domain gaps. Therefore, we propose a training-free
pipeline employing a pre-trained diffusion model imbued with semantic prior
knowledge, which can process composite videos with broader semantic
disparities. Specifically, we process the video frames in a cascading manner
and handle each frame in two processes with the diffusion model. In the
inversion process, we propose Balanced Partial Inversion to obtain generation
initial points that balance reversibility and modifiability. Then, in the
generation process, we further propose Inter-Frame Augmented attention to
augment foreground continuity across frames. Experimental results reveal that
our pipeline successfully ensures the visual harmony and inter-frame coherence
of the outputs, demonstrating efficacy in managing broader semantic
disparities.

通过使用一个预训练的扩散模型，处理具有更大语义差异的复合视频，我们的研究提出了一个无需训练的流程，以确保输出的视觉协调性和跨帧连贯性。