Diffusion Models (DMs) utilize an iterative denoising process to transform
random noise into synthetic data. Initally proposed with a UNet structure, DMs
excel at producing images that are virtually indistinguishable with or without
conditioned text prompts. Later transformer-only structure is composed with DMs
to achieve better performance. Though Latent Diffusion Models (LDMs) reduce the
computational requirement by denoising in a latent space, it is extremely
expensive to inference images for any operating devices due to the shear volume
of parameters and feature sizes. Post Training Quantization (PTQ) offers an
immediate remedy for a smaller storage size and more memory-efficient
computation during inferencing. Prior works address PTQ of DMs on UNet
structures have addressed the challenges in calibrating parameters for both
activations and weights via moderate optimization. In this work, we pioneer an
efficient PTQ on transformer-only structure without any optimization. By
analysing challenges in quantizing activations and weights for diffusion
transformers, we propose a single-step sampling calibration on activations and
adapt group-wise quantization on weights for low-bit quantization. We
demonstrate the efficiency and effectiveness of proposed methods with
preliminary experiments on conditional image generation.

通过分析扩散变换器中激活和权重量化的挑战，我们提出了一种单步采样校准激活和适应分组量化权重的低比特量化方法，从而在无需任何优化的情况下在纯变换器结构上实现了高效的后量化，我们通过初步的条件图像生成实验展示了所提出方法的效率和有效性。

量化扩散变压器分析

An Analysis on Quantizing Diffusion Transformers

Diffusion models have achieved significant visual generation quality.
However, their significant computational and memory costs pose challenge for
their application on resource-constrained mobile devices or even desktop GPUs.
Recent few-step diffusion models reduces the inference time by reducing the
denoising steps. However, their memory consumptions are still excessive. The
Post Training Quantization (PTQ) replaces high bit-width FP representation with
low-bit integer values (INT4/8) , which is an effective and efficient technique
to reduce the memory cost. However, when applying to few-step diffusion models,
existing quantization methods face challenges in preserving both the image
quality and text alignment. To address this issue, we propose an
mixed-precision quantization framework - MixDQ. Firstly, We design specialized
BOS-aware quantization method for highly sensitive text embedding quantization.
Then, we conduct metric-decoupled sensitivity analysis to measure the
sensitivity of each layer. Finally, we develop an integer-programming-based
method to conduct bit-width allocation. While existing quantization methods
fall short at W8A8, MixDQ could achieve W8A8 without performance loss, and W4A8
with negligible visual degradation. Compared with FP16, we achieve 3-4x
reduction in model size and memory cost, and 1.45x latency speedup.

通过开发一种混合精度量化框架 MixDQ，我们针对强敏感度文本嵌入进行专门的 BOS 感知量化方法设计，通过度量解耦灵敏度分析来衡量每一层的敏感度，最后通过基于整数规划的方法进行位宽分配，与现有的量化方法相比，MixDQ 在保持 W8A8 品质的同时，实现了模型大小和内存成本的 3-4 倍减少，和 1.45 倍的延迟加速。

MixDQ: 高效节省内存的几步文本到图像扩散模型与度量分离混合精度量化

MixDQ: Memory-Efficient Few-Step Text-to-Image Diffusion Models with  Metric-Decoupled Mixed Precision Quantization

Latent Diffusion Models (LDMs) capture the dynamic evolution of latent
variables over time, blending patterns and multimodality in a generative
system. Despite the proficiency of LDM in various applications, such as
text-to-image generation, facilitated by robust text encoders and a variational
autoencoder, the critical need to deploy large generative models on edge
devices compels a search for more compact yet effective alternatives. Post
Training Quantization (PTQ), a method to compress the operational size of deep
learning models, encounters challenges when applied to LDM due to temporal and
structural complexities. This study proposes a quantization strategy that
efficiently quantize LDMs, leveraging Signal-to-Quantization-Noise Ratio (SQNR)
as a pivotal metric for evaluation. By treating the quantization discrepancy as
relative noise and identifying sensitive part(s) of a model, we propose an
efficient quantization approach encompassing both global and local strategies.
The global quantization process mitigates relative quantization noise by
initiating higher-precision quantization on sensitive blocks, while local
treatments address specific challenges in quantization-sensitive and
time-sensitive modules. The outcomes of our experiments reveal that the
implementation of both global and local treatments yields a highly efficient
and effective Post Training Quantization (PTQ) of LDMs.

本研究提出了一种高效的量化策略，利用信噪比（SQNR）作为评估指标，将相对量化噪声视为噪声，识别模型中的敏感部分，并提出了既包括全局策略又包括局部策略的高效量化方法。实验结果表明，实施全局和局部策略能够高效地量化并有效地压缩 LDMs。