Denoising Diffusion models have exhibited remarkable capabilities in image generation. However, generating high-quality samples requires a large number of iterations. Knowledge distillation for diffusion models is an effective method to address this limitation with a shortened sampling process but causes degraded generative quality. Based on our analysis with bias-variance decomposition and experimental observations, we attribute the degradation to the spatial fitting error occurring in the training of both the teacher and student model. Accordingly, we propose $\textbf{S}$patial $\textbf{F}$itting-$\textbf{E}$rror $\textbf{R}$eduction $\textbf{D}$istillation model ($\textbf{SFERD}$). SFERD utilizes attention guidance from the teacher model and a designed semantic gradient predictor to reduce the student's fitting error. Empirically, our proposed model facilitates high-quality sample generation in a few function evaluations. We achieve an FID of 5.31 on CIFAR-10 and 9.39 on ImageNet 64$\times$64 with only one step, outperforming existing diffusion methods. Our study provides a new perspective on diffusion distillation by highlighting the intrinsic denoising ability of models.

通过分析偏差-方差分解和实验观察，我们提出了一种基于空间拟合误差减少的蒸馏模型（SFERD），该模型利用教师模型的注意力引导和设计的语义梯度预测器来降低学生模型的拟合误差，从而在少量函数评价中实现高质量的样本生成，并在CIFAR-10上取得了5.31的FID和ImageNet 64×64上取得了9.39的FID，超过了现有的扩散方法。我们的研究突出了模型的本质去噪能力，为扩散蒸馏提供了新的视角。

减少噪声扩散模型蒸馏中的空间拟合误差