Video editing methods based on diffusion models that rely solely on a text
prompt for the edit are hindered by the limited expressive power of text
prompts. Thus, incorporating a reference target image as a visual guide becomes
desirable for precise control over edit. Also, most existing methods struggle
to accurately edit a video when the shape and size of the object in the target
image differ from the source object. To address these challenges, we propose
"GenVideo" for editing videos leveraging target-image aware T2I models. Our
approach handles edits with target objects of varying shapes and sizes while
maintaining the temporal consistency of the edit using our novel target and
shape aware InvEdit masks. Further, we propose a novel target-image aware
latent noise correction strategy during inference to improve the temporal
consistency of the edits. Experimental analyses indicate that GenVideo can
effectively handle edits with objects of varying shapes, where existing
approaches fail.

提出了一种名为 “GenVideo” 的视频编辑方法，它利用目标图像感知的 T2I 模型进行编辑，能够处理具有不同形状的目标对象的编辑，并通过新颖的目标和形状感知 InvEdit 蒙版保持编辑的时间一致性。实验分析表明，GenVideo 能够有效处理以往方法无法处理的具有不同形状的对象的编辑。

GenVideo: 利用 T2I 扩散模型进行单次目标图像和形状感知视频编辑

GenVideo: One-shot Target-image and Shape Aware Video Editing using T2I  Diffusion Models

Current diffusion-based image restoration methods feed degraded input images
as conditions into the noise estimation network. However, interpreting this
diffusion process is challenging since it essentially generates the target
image from the noise. To establish a unified and more interpretable model for
image generation and restoration, we propose residual denoising diffusion
models (RDDM). In contrast to existing diffusion models (e.g., DDPM or DDIM)
that focus solely on noise estimation, our RDDM predicts residuals to represent
directional diffusion from the target domain to the input domain, while
concurrently estimating noise to account for random perturbations in the
diffusion process. The introduction of residuals allows us to redefine the
forward diffusion process, wherein the target image progressively diffuses into
a purely noisy image or a noise-carrying input image, thus unifying image
generation and restoration. We demonstrate that our sampling process is
consistent with that of DDPM and DDIM through coefficient transformation, and
propose a partially path-independent generation process to better understand
the reverse process. Notably, with native support for conditional inputs, our
RDDM enables a generic UNet, trained with only an $\ell _1$ loss and a batch
size of 1, to compete with state-of-the-art image restoration methods. We
provide code and pre-trained models to encourage further exploration,
application, and development of our innovative framework
(this https URL).

提出了一种基于残差去噪扩散模型（RDDM）的图像生成和恢复方法，该方法通过预测残差来表示从目标域到输入域的扩散方向，并同时估计噪声以考虑扩散过程中的随机扰动，从而实现了图像生成和恢复的统一。