Text-conditioned image-to-video generation (TI2V) aims to synthesize a
realistic video starting from a given image (e.g., a woman's photo) and a text
description (e.g., "a woman is drinking water."). Existing TI2V frameworks
often require costly training on video-text datasets and specific model designs
for text and image conditioning. In this paper, we propose TI2V-Zero, a
zero-shot, tuning-free method that empowers a pretrained text-to-video (T2V)
diffusion model to be conditioned on a provided image, enabling TI2V generation
without any optimization, fine-tuning, or introducing external modules. Our
approach leverages a pretrained T2V diffusion foundation model as the
generative prior. To guide video generation with the additional image input, we
propose a "repeat-and-slide" strategy that modulates the reverse denoising
process, allowing the frozen diffusion model to synthesize a video
frame-by-frame starting from the provided image. To ensure temporal continuity,
we employ a DDPM inversion strategy to initialize Gaussian noise for each newly
synthesized frame and a resampling technique to help preserve visual details.
We conduct comprehensive experiments on both domain-specific and open-domain
datasets, where TI2V-Zero consistently outperforms a recent open-domain TI2V
model. Furthermore, we show that TI2V-Zero can seamlessly extend to other tasks
such as video infilling and prediction when provided with more images. Its
autoregressive design also supports long video generation.

提出了 TI2V-Zero：一种无需优化或微调的零样本方法，通过使用预训练的文本到视频扩散模型，使其能够在给定图像的条件下生成实际视频。该方法使用 “重复滑动” 策略来引导视频生成，并提供了一种保持视觉细节的新合成帧初始化和重新采样技术，从而实现视频的逐帧合成。TI2V-Zero 在领域特定和开放领域数据集上的实验证明了其优越性能，并且能够无缝扩展到其他任务和支持长视频生成。