Text-driven diffusion models have unlocked unprecedented abilities in image
generation, whereas their video counterpart still lags behind due to the
excessive training cost of temporal modeling. Besides the training burden, the
generated videos also suffer from appearance inconsistency and structural
flickers, especially in long video synthesis. To address these challenges, we
design a \emph{training-free} framework called \textbf{ControlVideo} to enable
natural and efficient text-to-video generation. ControlVideo, adapted from
ControlNet, leverages coarsely structural consistency from input motion
sequences, and introduces three modules to improve video generation. Firstly,
to ensure appearance coherence between frames, ControlVideo adds fully
cross-frame interaction in self-attention modules. Secondly, to mitigate the
flicker effect, it introduces an interleaved-frame smoother that employs frame
interpolation on alternated frames. Finally, to produce long videos
efficiently, it utilizes a hierarchical sampler that separately synthesizes
each short clip with holistic coherency. Empowered with these modules,
ControlVideo outperforms the state-of-the-arts on extensive motion-prompt pairs
quantitatively and qualitatively. Notably, thanks to the efficient designs, it
generates both short and long videos within several minutes using one NVIDIA
2080Ti. Code is available at this https URL

ControlVideo 是一种基于文本驱动扩散模型的、无需训练即可完成自然高效文本到视频生成，使用三个模块实现外观协调、帧插值以及分层采样，能够在轻松几分钟之内生成高质量的短视频和长视频。