The rise of billion-parameter diffusion models like Stable Diffusion XL,
Imagen, and Dall-E3 markedly advances the field of generative AI. However,
their large-scale nature poses challenges in fine-tuning and deployment due to
high resource demands and slow inference speed. This paper ventures into the
relatively unexplored yet promising realm of fine-tuning quantized diffusion
models. We establish a strong baseline by customizing three models: PEQA for
fine-tuning quantization parameters, Q-Diffusion for post-training
quantization, and DreamBooth for personalization. Our analysis reveals a
notable trade-off between subject and prompt fidelity within the baseline
model. To address these issues, we introduce two strategies, inspired by the
distinct roles of different timesteps in diffusion models: S1 optimizing a
single set of fine-tuning parameters exclusively at selected intervals, and S2
creating multiple fine-tuning parameter sets, each specialized for different
timestep intervals. Our approach not only enhances personalization but also
upholds prompt fidelity and image quality, significantly outperforming the
baseline qualitatively and quantitatively. The code will be made publicly
available.

本文针对大规模的十亿参数扩散模型，探索了细调量化扩散模型的领域，并提出了两种策略以增强个性化、保持提示保真度和图像质量，显著超越基线模型的质量和数量性能。