With the advancement of diffusion models (DMs) and the substantially
increased computational requirements, quantization emerges as a practical
solution to obtain compact and efficient low-bit DMs. However, the highly
discrete representation leads to severe accuracy degradation, hindering the
quantization of diffusion models to ultra-low bit-widths. In this paper, we
propose BinaryDM, a novel accurate quantization-aware training approach to push
the weights of diffusion models towards the limit of 1-bit. Firstly, we present
a Learnable Multi-basis Binarizer (LMB) to recover the representations
generated by the binarized DM, which improves the information in details of
representations crucial to the DM. Secondly, a Low-rank Representation
Mimicking (LRM) is applied to enhance the binarization-aware optimization of
the DM, alleviating the optimization direction ambiguity caused by fine-grained
alignment. Moreover, a progressive initialization strategy is applied to
training DMs to avoid convergence difficulties. Comprehensive experiments
demonstrate that BinaryDM achieves significant accuracy and efficiency gains
compared to SOTA quantization methods of DMs under ultra-low bit-widths. As the
first binarization method for diffusion models, BinaryDM achieves impressive
16.0 times FLOPs and 27.1 times storage savings with 1-bit weight and 4-bit
activation, showcasing its substantial advantages and potential for deploying
DMs on resource-limited scenarios.

本研究提出了 BinaryDM，一种新颖的准确的量化感知训练方法，以将扩散模型的权重推向 1 位极限。使用 Learnable Multi-basis Binarizer (LMB) 来恢复二元化扩散模型生成的表示，并使用 Low-rank Representation Mimicking (LRM) 来提高二元化感知优化。通过渐进的初始化策略训练扩散模型，可以避免收敛困难。全面的实验证明，与超低位宽下的 DM 的 SOTA 量化方法相比，BinaryDM 在精度和效率上都取得了显著的改进。作为扩散模型的第一种二元化方法，BinaryDM 在具有 1 位权重和 4 位激活的情况下可实现 16.0 倍的 FLOPs 和 27.1 倍的存储节省，展示了其在资源有限场景中的巨大优势和潜力。