With the advancement of diffusion models (DMs) and the substantially
increased computational requirements, quantization emerges as a practical
solution to obtain compact and efficient low-bit DMs. However, the highly
discrete representation leads to severe accuracy degradation, hindering the
quantization of diffusion models to ultra-low bit-widths. In this paper, we
propose BinaryDM, a novel accurate quantization-aware training approach to push
the weights of diffusion models towards the limit of 1-bit. Firstly, we present
a Learnable Multi-basis Binarizer (LMB) to recover the representations
generated by the binarized DM, which improves the information in details of
representations crucial to the DM. Secondly, a Low-rank Representation
Mimicking (LRM) is applied to enhance the binarization-aware optimization of
the DM, alleviating the optimization direction ambiguity caused by fine-grained
alignment. Moreover, a progressive initialization strategy is applied to
training DMs to avoid convergence difficulties. Comprehensive experiments
demonstrate that BinaryDM achieves significant accuracy and efficiency gains
compared to SOTA quantization methods of DMs under ultra-low bit-widths. As the
first binarization method for diffusion models, BinaryDM achieves impressive
16.0 times FLOPs and 27.1 times storage savings with 1-bit weight and 4-bit
activation, showcasing its substantial advantages and potential for deploying
DMs on resource-limited scenarios.

本研究提出了 BinaryDM，一种新颖的准确的量化感知训练方法，以将扩散模型的权重推向 1 位极限。使用 Learnable Multi-basis Binarizer (LMB) 来恢复二元化扩散模型生成的表示，并使用 Low-rank Representation Mimicking (LRM) 来提高二元化感知优化。通过渐进的初始化策略训练扩散模型，可以避免收敛困难。全面的实验证明，与超低位宽下的 DM 的 SOTA 量化方法相比，BinaryDM 在精度和效率上都取得了显著的改进。作为扩散模型的第一种二元化方法，BinaryDM 在具有 1 位权重和 4 位激活的情况下可实现 16.0 倍的 FLOPs 和 27.1 倍的存储节省，展示了其在资源有限场景中的巨大优势和潜力。

BinaryDM: 混合扩散模型的准确二值化

BinaryDM: Towards Accurate Binarization of Diffusion Model

The deep neural networks, such as the Deep-FSMN, have been widely studied for
keyword spotting (KWS) applications. However, computational resources for these
networks are significantly constrained since they usually run on-call on edge
devices. In this paper, we present BiFSMN, an accurate and extreme-efficient
binary neural network for KWS. We first construct a High-frequency Enhancement
Distillation scheme for the binarization-aware training, which emphasizes the
high-frequency information from the full-precision network's representation
that is more crucial for the optimization of the binarized network. Then, to
allow the instant and adaptive accuracy-efficiency trade-offs at runtime, we
also propose a Thinnable Binarization Architecture to further liberate the
acceleration potential of the binarized network from the topology perspective.
Moreover, we implement a Fast Bitwise Computation Kernel for BiFSMN on ARMv8
devices which fully utilizes registers and increases instruction throughput to
push the limit of deployment efficiency. Extensive experiments show that BiFSMN
outperforms existing binarization methods by convincing margins on various
datasets and is even comparable with the full-precision counterpart (e.g., less
than 3% drop on Speech Commands V1-12). We highlight that benefiting from the
thinnable architecture and the optimized 1-bit implementation, BiFSMN can
achieve an impressive 22.3x speedup and 15.5x storage-saving on real-world edge
hardware. Our code is released at this https URL.

本研究提出了 BiFSMN，一种精度高、效率极高的二进制神经网络，它包括了针对一些关键信息的深度蒸馏机制、一种优化的二进制化训练方法、可以自适应提高通用性和效率的纤薄化结构和一个在 ARMv8 设备上的快速位运算核心，实现了在各种数据集上完备性和效率上的优化。 BiFSMN 可以在实际硬件中获得惊人的 22.3 倍加速和 15.5 倍存储空间节省。