We present an efficient text-to-video generation framework based on latent
diffusion models, termed MagicVideo. Given a text description, MagicVideo can
generate photo-realistic video clips with high relevance to the text content.
With the proposed efficient latent 3D U-Net design, MagicVideo can generate
video clips with 256x256 spatial resolution on a single GPU card, which is 64x
faster than the recent video diffusion model (VDM). Unlike previous works that
train video generation from scratch in the RGB space, we propose to generate
video clips in a low-dimensional latent space. We further utilize all the
convolution operator weights of pre-trained text-to-image generative U-Net
models for faster training. To achieve this, we introduce two new designs to
adapt the U-Net decoder to video data: a framewise lightweight adaptor for the
image-to-video distribution adjustment and a directed temporal attention module
to capture frame temporal dependencies. The whole generation process is within
the low-dimension latent space of a pre-trained variation auto-encoder. We
demonstrate that MagicVideo can generate both realistic video content and
imaginary content in a photo-realistic style with a trade-off in terms of
quality and computational cost. Refer to this https URL for
more examples.

MagicVideo 利用基于潜在扩散模型的高效文本生成视频框架生成与文本内容高度相关的逼真视频片段，并使用低维潜在空间在单个 GPU 卡上生成具有 256x256 空间分辨率的视频片段，可以在 64 倍于最近的视频扩散模型（VDM）的速度下生成。