Diffusion Probability Models (DPMs) have made impressive advancements in
various machine learning domains. However, achieving high-quality synthetic
samples typically involves performing a large number of sampling steps, which
impedes the possibility of real-time sample synthesis. Traditional accelerated
sampling algorithms via knowledge distillation rely on pre-trained model
weights and discrete time step scenarios, necessitating additional training
sessions to achieve their goals. To address these issues, we propose the
Catch-Up Distillation (CUD), which encourages the current moment output of the
velocity estimation model ``catch up'' with its previous moment output.
Specifically, CUD adjusts the original Ordinary Differential Equation (ODE)
training objective to align the current moment output with both the ground
truth label and the previous moment output, utilizing Runge-Kutta-based
multi-step alignment distillation for precise ODE estimation while preventing
asynchronous updates. Furthermore, we investigate the design space for CUDs
under continuous time-step scenarios and analyze how to determine the suitable
strategies. To demonstrate CUD's effectiveness, we conduct thorough ablation
and comparison experiments on CIFAR-10, MNIST, and ImageNet-64. On CIFAR-10, we
obtain a FID of 2.80 by sampling in 15 steps under one-session training and the
new state-of-the-art FID of 3.37 by sampling in one step with additional
training. This latter result necessitated only 62w iterations with a batch size
of 128, in contrast to Consistency Distillation, which demanded 210w iterations
with a larger batch size of 256.

为了解决 Diffusion Probability Models 生成样本所需的采样步骤大量增加的问题，文章提出了一种基于 Catch-Up Distillation (CUD) 的算法，通过多步对齐蒙古采样输出来实现精确 ODE 估计，成功地在 CIFAR-10，MNIST 和 ImageNet-64 数据集上取得了比较优异的效果。