The recent wave of AI-generated content has witnessed the great development
and success of Text-to-Image (T2I) technologies. By contrast, Text-to-Video
(T2V) still falls short of expectations though attracting increasing interests.
Existing works either train from scratch or adapt large T2I model to videos,
both of which are computation and resource expensive. In this work, we propose
a Simple Diffusion Adapter (SimDA) that fine-tunes only 24M out of 1.1B
parameters of a strong T2I model, adapting it to video generation in a
parameter-efficient way. In particular, we turn the T2I model for T2V by
designing light-weight spatial and temporal adapters for transfer learning.
Besides, we change the original spatial attention to the proposed Latent-Shift
Attention (LSA) for temporal consistency. With similar model architecture, we
further train a video super-resolution model to generate high-definition
(1024x1024) videos. In addition to T2V generation in the wild, SimDA could also
be utilized in one-shot video editing with only 2 minutes tuning. Doing so, our
method could minimize the training effort with extremely few tunable parameters
for model adaptation.

本文提出了一种 Simple Diffusion Adapter (SimDA) 方法，通过微调一个强大的 Text-to-Image (T2I) 模型的 24M 个参数，以高效的方式将其适应于视频生成，并使用轻量级的空间和时间适配器进行转移学习，为了达到视频的一致性还提出了一种新的潜在干扰 Attention (LSA) 方法。此外，我们还使用相似的模型架构训练了一个视频超分辨率模型，以生成高清 (1024x1024) 视频。SimDA 不仅可用于野外的 T2V 生成，还能够在 2 分钟的调整中实现一键视频编辑，以极少的可调参数来减少训练工作量。