Recent works have successfully extended large-scale text-to-image models to
the video domain, producing promising results but at a high computational cost
and requiring a large amount of video data. In this work, we introduce
ConditionVideo, a training-free approach to text-to-video generation based on
the provided condition, video, and input text, by leveraging the power of
off-the-shelf text-to-image generation methods (e.g., Stable Diffusion).
ConditionVideo generates realistic dynamic videos from random noise or given
scene videos. Our method explicitly disentangles the motion representation into
condition-guided and scenery motion components. To this end, the ConditionVideo
model is designed with a UNet branch and a control branch. To improve temporal
coherence, we introduce sparse bi-directional spatial-temporal attention
(sBiST-Attn). The 3D control network extends the conventional 2D controlnet
model, aiming to strengthen conditional generation accuracy by additionally
leveraging the bi-directional frames in the temporal domain. Our method
exhibits superior performance in terms of frame consistency, clip score, and
conditional accuracy, outperforming other compared methods.

在本文中，我们介绍了一种无需训练的文本到视频生成方法，通过利用现有的文本到图像生成方法生成逼真的动态视频，这种方法以条件、视频和输入文本为基础。我们的方法将运动表示明确地分为条件引导和场景运动组成，并引入了稀疏双向时空注意力来改善时序一致性。与其他比较方法相比，我们的方法在帧一致性、剪辑评分和条件精度方面表现出卓越性能。