We present Pix2Gif, a motion-guided diffusion model for image-to-GIF (video)
generation. We tackle this problem differently by formulating the task as an
image translation problem steered by text and motion magnitude prompts, as
shown in teaser fig. To ensure that the model adheres to motion guidance, we
propose a new motion-guided warping module to spatially transform the features
of the source image conditioned on the two types of prompts. Furthermore, we
introduce a perceptual loss to ensure the transformed feature map remains
within the same space as the target image, ensuring content consistency and
coherence. In preparation for the model training, we meticulously curated data
by extracting coherent image frames from the TGIF video-caption dataset, which
provides rich information about the temporal changes of subjects. After
pretraining, we apply our model in a zero-shot manner to a number of video
datasets. Extensive qualitative and quantitative experiments demonstrate the
effectiveness of our model -- it not only captures the semantic prompt from
text but also the spatial ones from motion guidance. We train all our models
using a single node of 16xV100 GPUs. Code, dataset and models are made public
at: this https URL

我们提出了 Pix2Gif，一种用于图像到 GIF（视频）生成的运动引导扩散模型。通过将任务表述为由文本和运动幅度提示引导的图像转换问题，我们以不同的方式解决了这个问题。为了确保模型遵循运动引导，我们提出了一种新的运动引导变形模块，以在两种提示的条件下空间转换源图像的特征。此外，我们引入了一个感知损失，确保变换后的特征图与目标图像保持相同的空间，以确保内容的一致性和连贯性。在模型训练准备阶段，我们通过从 TGIF 视频标题数据集中提取一致的图像帧来精心筛选数据，该数据集提供了关于主题的时序变化的丰富信息。在预训练之后，我们以零样本的方式将我们的模型应用于多个视频数据集。广泛的定性和定量实验证明了我们的模型的有效性 - 它不仅捕捉了文本的语义提示，还捕捉了运动引导的空间提示。我们使用一个 16xV100 GPU 节点训练所有模型。代码、数据集和模型在此 https URL 中公开。