Diffusion models have demonstrated remarkable capabilities in image synthesis
and related generative tasks. Nevertheless, their practicality for low-latency
real-world applications is constrained by substantial computational costs and
latency issues. Quantization is a dominant way to compress and accelerate
diffusion models, where post-training quantization (PTQ) and quantization-aware
training (QAT) are two main approaches, each bearing its own properties. While
PTQ exhibits efficiency in terms of both time and data usage, it may lead to
diminished performance in low bit-width. On the other hand, QAT can alleviate
performance degradation but comes with substantial demands on computational and
data resources. To capitalize on the advantages while avoiding their respective
drawbacks, we introduce a data-free and parameter-efficient fine-tuning
framework for low-bit diffusion models, dubbed EfficientDM, to achieve
QAT-level performance with PTQ-like efficiency. Specifically, we propose a
quantization-aware variant of the low-rank adapter (QALoRA) that can be merged
with model weights and jointly quantized to low bit-width. The fine-tuning
process distills the denoising capabilities of the full-precision model into
its quantized counterpart, eliminating the requirement for training data. We
also introduce scale-aware optimization and employ temporal learned step-size
quantization to further enhance performance. Extensive experimental results
demonstrate that our method significantly outperforms previous PTQ-based
diffusion models while maintaining similar time and data efficiency.
Specifically, there is only a marginal 0.05 sFID increase when quantizing both
weights and activations of LDM-4 to 4-bit on ImageNet 256x256. Compared to
QAT-based methods, our EfficientDM also boasts a 16.2x faster quantization
speed with comparable generation quality.

提出了一种数据自由且参数高效的优化方法 EfficientDM，它通过引入量化感知的低秩适配器 QALoRA 来实现 QAT 级别的性能与 PTQ 的效率相似，从而优化低位扩散模型的性能，并且在时间和数据效率方面显著超过之前基于 PTQ 的扩散模型。