Diffusion models have recently dominated image synthesis and other related
generative tasks. However, the iterative denoising process is expensive in
computations at inference time, making diffusion models less practical for
low-latency and scalable real-world applications. Post-training quantization of
diffusion models can significantly reduce the model size and accelerate the
sampling process without requiring any re-training. Nonetheless, applying
existing post-training quantization methods directly to low-bit diffusion
models can significantly impair the quality of generated samples. Specifically,
for each denoising step, quantization noise leads to deviations in the
estimated mean and mismatches with the predetermined variance schedule.
Moreover, as the sampling process proceeds, the quantization noise may
accumulate, resulting in a low signal-to-noise ratio (SNR) in late denoising
steps. To address these challenges, we propose a unified formulation for the
quantization noise and diffusion perturbed noise in the quantized denoising
process. We first disentangle the quantization noise into its correlated and
residual uncorrelated parts regarding its full-precision counterpart. The
correlated part can be easily corrected by estimating the correlation
coefficient. For the uncorrelated part, we calibrate the denoising variance
schedule to absorb the excess variance resulting from quantization. Moreover,
we propose a mixed-precision scheme to choose the optimal bitwidth for each
denoising step, which prefers low bits to accelerate the early denoising steps
while high bits maintain the high SNR for the late steps. Extensive experiments
demonstrate that our method outperforms previous post-training quantized
diffusion models in generating high-quality samples, with only a 0.06 increase
in FID score compared to full-precision LDM-4 on ImageNet 256x256, while saving
19.9x bit operations.

本研究提出了一种用于量化噪声和扩散扰动噪声的统一方法，并使用所提出的混合精度方案，校准降噪方差表并选取每个降噪步骤的最佳位宽，显著提高了量化后扩散模型的样本质量且减少了操作次数。