Text-to-image diffusion models (T2I) have demonstrated unprecedented
capabilities in creating realistic and aesthetic images. On the contrary,
text-to-video diffusion models (T2V) still lag far behind in frame quality and
text alignment, owing to insufficient quality and quantity of training videos.
In this paper, we introduce VideoElevator, a training-free and plug-and-play
method, which elevates the performance of T2V using superior capabilities of
T2I. Different from conventional T2V sampling (i.e., temporal and spatial
modeling), VideoElevator explicitly decomposes each sampling step into temporal
motion refining and spatial quality elevating. Specifically, temporal motion
refining uses encapsulated T2V to enhance temporal consistency, followed by
inverting to the noise distribution required by T2I. Then, spatial quality
elevating harnesses inflated T2I to directly predict less noisy latent, adding
more photo-realistic details. We have conducted experiments in extensive
prompts under the combination of various T2V and T2I. The results show that
VideoElevator not only improves the performance of T2V baselines with
foundational T2I, but also facilitates stylistic video synthesis with
personalized T2I. Our code is available at
this https URL

Text-to-video diffusion models still lag behind compared to text-to-image models, so this paper introduces VideoElevator, a training-free and plug-and-play method that improves T2V performance using T2I capabilities, specifically by enhancing temporal consistency and adding photo-realistic details for personalized stylistic video synthesis.

视频电梯：用多功能的文本到图像扩散模型提升视频生成质量

VideoElevator: Elevating Video Generation Quality with Versatile  Text-to-Image Diffusion Models

The recent wave of AI-generated content has witnessed the great development
and success of Text-to-Image (T2I) technologies. By contrast, Text-to-Video
(T2V) still falls short of expectations though attracting increasing interests.
Existing works either train from scratch or adapt large T2I model to videos,
both of which are computation and resource expensive. In this work, we propose
a Simple Diffusion Adapter (SimDA) that fine-tunes only 24M out of 1.1B
parameters of a strong T2I model, adapting it to video generation in a
parameter-efficient way. In particular, we turn the T2I model for T2V by
designing light-weight spatial and temporal adapters for transfer learning.
Besides, we change the original spatial attention to the proposed Latent-Shift
Attention (LSA) for temporal consistency. With similar model architecture, we
further train a video super-resolution model to generate high-definition
(1024x1024) videos. In addition to T2V generation in the wild, SimDA could also
be utilized in one-shot video editing with only 2 minutes tuning. Doing so, our
method could minimize the training effort with extremely few tunable parameters
for model adaptation.

本文提出了一种 Simple Diffusion Adapter (SimDA) 方法，通过微调一个强大的 Text-to-Image (T2I) 模型的 24M 个参数，以高效的方式将其适应于视频生成，并使用轻量级的空间和时间适配器进行转移学习，为了达到视频的一致性还提出了一种新的潜在干扰 Attention (LSA) 方法。此外，我们还使用相似的模型架构训练了一个视频超分辨率模型，以生成高清 (1024x1024) 视频。SimDA 不仅可用于野外的 T2V 生成，还能够在 2 分钟的调整中实现一键视频编辑，以极少的可调参数来减少训练工作量。

SimDA：用于高效视频生成的简易扩散适配器

SimDA: Simple Diffusion Adapter for Efficient Video Generation

We propose Make-A-Video -- an approach for directly translating the
tremendous recent progress in Text-to-Image (T2I) generation to Text-to-Video
(T2V). Our intuition is simple: learn what the world looks like and how it is
described from paired text-image data, and learn how the world moves from
unsupervised video footage. Make-A-Video has three advantages: (1) it
accelerates training of the T2V model (it does not need to learn visual and
multimodal representations from scratch), (2) it does not require paired
text-video data, and (3) the generated videos inherit the vastness (diversity
in aesthetic, fantastical depictions, etc.) of today's image generation models.
We design a simple yet effective way to build on T2I models with novel and
effective spatial-temporal modules. First, we decompose the full temporal U-Net
and attention tensors and approximate them in space and time. Second, we design
a spatial temporal pipeline to generate high resolution and frame rate videos
with a video decoder, interpolation model and two super resolution models that
can enable various applications besides T2V. In all aspects, spatial and
temporal resolution, faithfulness to text, and quality, Make-A-Video sets the
new state-of-the-art in text-to-video generation, as determined by both
qualitative and quantitative measures.

该研究提出一种名为 Make-A-Video 的方法，利用文本 - 图像生成的进展，从而实现文本 - 视频的生成，通过对时空模块的研究，提高了空间和时间的分辨率和文本的保真度并取得了最新的成果。