Diffusion models are the de facto approach for generating high-quality images
and videos, but learning high-dimensional models remains a formidable task due
to computational and optimization challenges. Existing methods often resort to
training cascaded models in pixel space or using a downsampled latent space of
a separately trained auto-encoder. In this paper, we introduce Matryoshka
Diffusion Models(MDM), an end-to-end framework for high-resolution image and
video synthesis. We propose a diffusion process that denoises inputs at
multiple resolutions jointly and uses a NestedUNet architecture where features
and parameters for small-scale inputs are nested within those of large scales.
In addition, MDM enables a progressive training schedule from lower to higher
resolutions, which leads to significant improvements in optimization for
high-resolution generation. We demonstrate the effectiveness of our approach on
various benchmarks, including class-conditioned image generation,
high-resolution text-to-image, and text-to-video applications. Remarkably, we
can train a single pixel-space model at resolutions of up to 1024x1024 pixels,
demonstrating strong zero-shot generalization using the CC12M dataset, which
contains only 12 million images.

引入了 Matryoshka Diffusion Models（MDM），这是一种用于高分辨率图像和视频合成的端到端框架。通过在多个分辨率上联合去噪输入，使用嵌套 UNet 架构和从低到高分辨率的渐进式训练安排，实现了高分辨率生成的显著优化改进。在各种基准测试中展示了该方法的有效性，包括类别条件图像生成、高分辨率文本到图像以及文本到视频应用。在仅包含 1200 万张图像的 CC12M 数据集上，我们可以训练一个单一像素空间模型，分辨率达到 1024x1024 像素，并展现了强大的零样本泛化能力。