Text-to-video diffusion models enable the generation of high-quality videos
that follow text instructions, making it easy to create diverse and individual
content. However, existing approaches mostly focus on high-quality short video
generation (typically 16 or 24 frames), ending up with hard-cuts when naively
extended to the case of long video synthesis. To overcome these limitations, we
introduce StreamingT2V, an autoregressive approach for long video generation of
80, 240, 600, 1200 or more frames with smooth transitions. The key components
are:(i) a short-term memory block called conditional attention module (CAM),
which conditions the current generation on the features extracted from the
previous chunk via an attentional mechanism, leading to consistent chunk
transitions, (ii) a long-term memory block called appearance preservation
module, which extracts high-level scene and object features from the first
video chunk to prevent the model from forgetting the initial scene, and (iii) a
randomized blending approach that enables to apply a video enhancer
autoregressively for infinitely long videos without inconsistencies between
chunks. Experiments show that StreamingT2V generates high motion amount. In
contrast, all competing image-to-video methods are prone to video stagnation
when applied naively in an autoregressive manner. Thus, we propose with
StreamingT2V a high-quality seamless text-to-long video generator that
outperforms competitors with consistency and motion. Our code will be available
at: this https URL

通过 StreamingT2V 方法，可以实现高质量的长视频生成，该方法采用文本到视频扩散模型并引入条件注意模块，具有一致性和高运动量。