Diffusion models have proven to be highly effective in image and video
generation; however, they still face composition challenges when generating
images of varying sizes due to single-scale training data. Adapting large
pre-trained diffusion models for higher resolution demands substantial
computational and optimization resources, yet achieving a generation capability
comparable to low-resolution models remains elusive. This paper proposes a
novel self-cascade diffusion model that leverages the rich knowledge gained
from a well-trained low-resolution model for rapid adaptation to
higher-resolution image and video generation, employing either tuning-free or
cheap upsampler tuning paradigms. Integrating a sequence of multi-scale
upsampler modules, the self-cascade diffusion model can efficiently adapt to a
higher resolution, preserving the original composition and generation
capabilities. We further propose a pivot-guided noise re-schedule strategy to
speed up the inference process and improve local structural details. Compared
to full fine-tuning, our approach achieves a 5X training speed-up and requires
only an additional 0.002M tuning parameters. Extensive experiments demonstrate
that our approach can quickly adapt to higher resolution image and video
synthesis by fine-tuning for just 10k steps, with virtually no additional
inference time.

本文提出了一种新颖的自级联扩散模型，通过利用低分辨率模型的丰富知识快速适应高分辨率图像和视频生成，使用无需调优或廉价上采样器调优范例。该自级联扩散模型整合了一系列多尺度上采样器模块，可以有效地适应更高的分辨率，保持原始组合和生成能力。我们进一步提出了一种基于中心引导的噪声重新调度策略，以加快推理过程并提高局部结构细节。与完全微调相比，我们的方法在训练速度上提高了 5 倍，并且仅需要额外的 0.002M 调优参数。广泛的实验表明，我们的方法可以通过仅仅进行 10k 步的微调，快速适应更高分辨率的图像和视频合成，几乎不需要额外的推理时间。