The goal of conditional image-to-video (cI2V) generation is to create a
believable new video by beginning with the condition, i.e., one image and
text.The previous cI2V generation methods conventionally perform in RGB pixel
space, with limitations in modeling motion consistency and visual continuity.
Additionally, the efficiency of generating videos in pixel space is quite
low.In this paper, we propose a novel approach to address these challenges by
disentangling the target RGB pixels into two distinct components: spatial
content and temporal motions. Specifically, we predict temporal motions which
include motion vector and residual based on a 3D-UNet diffusion model. By
explicitly modeling temporal motions and warping them to the starting image, we
improve the temporal consistency of generated videos. This results in a
reduction of spatial redundancy, emphasizing temporal details. Our proposed
method achieves performance improvements by disentangling content and motion,
all without introducing new structural complexities to the model. Extensive
experiments on various datasets confirm our approach's superior performance
over the majority of state-of-the-art methods in both effectiveness and
efficiency.

通过将目标 RGB 像素分解成空间内容和时间运动两个不同的组成部分，我们提出了一种解决传统 RGB 像素空间方法中涉及建模动作一致性和视觉连贯性限制的新方法。通过显式建模时间运动并将其应用于起始图像，我们改进了生成视频的时间一致性，减少了空间冗余，突出了时间细节。大量实验证实了我们的方法相对于大多数最先进的方法在效果和效率上的卓越性能。