Recent advancements in diffusion models, particularly the trend of
architectural transformation from UNet-based Diffusion to Diffusion Transformer
(DiT), have significantly improved the quality and scalability of image
synthesis. Despite the incredible generative quality, the large computational
requirements of these large-scale models significantly hinder the deployments
in real-world scenarios. Post-training Quantization (PTQ) offers a promising
solution by compressing model sizes and speeding up inference for the
pretrained models while eliminating model retraining. However, we have observed
the existing PTQ frameworks exclusively designed for both ViT and conventional
Diffusion models fall into biased quantization and result in remarkable
performance degradation. In this paper, we find that the DiTs typically exhibit
considerable variance in terms of both weight and activation, which easily runs
out of the limited numerical representations. To address this issue, we devise
Q-DiT, which seamlessly integrates three techniques: fine-grained quantization
to manage substantial variance across input channels of weights and
activations, an automatic search strategy to optimize the quantization
granularity and mitigate redundancies, and dynamic activation quantization to
capture the activation changes across timesteps. Extensive experiments on the
ImageNet dataset demonstrate the effectiveness of the proposed Q-DiT.
Specifically, when quantizing DiT-XL/2 to W8A8 on ImageNet 256x256, Q-DiT
achieves a remarkable reduction in FID by 1.26 compared to the baseline. Under
a W4A8 setting, it maintains high fidelity in image generation, showcasing only
a marginal increase in FID and setting a new benchmark for efficient,
high-quality quantization in diffusion transformers. Code is available at
\href{https://github.com/Juanerx/Q-DiT}{this https URL}.

Q-DiT 是一种结合了精细化量化、自动搜索策略和动态激活量化的方法，用于处理 Diffusion Transformer（DiT）模型中的权重和激活的巨大变化，以实现高效、高质量的量化和图像生成。

Q-DiT：扩散变压器的准确后训练量化

Q-DiT: Accurate Post-Training Quantization for Diffusion Transformers

Recently, the tokens of images share the same static data flow in many dense
networks. However, challenges arise from the variance among the objects in
images, such as large variations in the spatial scale and difficulties of
recognition for visual entities. In this paper, we propose a data-dependent
token routing strategy to elaborate the routing paths of image tokens for
Dynamic Vision Transformer, dubbed DiT. The proposed framework generates a
data-dependent path per token, adapting to the object scales and visual
discrimination of tokens. In feed-forward, the differentiable routing gates are
designed to select the scaling paths and feature transformation paths for image
tokens, leading to multi-path feature propagation. In this way, the impact of
object scales and visual discrimination of image representation can be
carefully tuned. Moreover, the computational cost can be further reduced by
giving budget constraints to the routing gate and early-stopping of feature
extraction. In experiments, our DiT achieves superior performance and favorable
complexity/accuracy trade-offs than many SoTA methods on ImageNet
classification, object detection, instance segmentation, and semantic
segmentation. Particularly, the DiT-B5 obtains 84.8\% top-1 Acc on ImageNet
with 10.3 GFLOPs, which is 1.0\% higher than that of the SoTA method with
similar computational complexity. These extensive results demonstrate that DiT
can serve as versatile backbones for various vision tasks.

提出了一种数据相关的令牌路由策略用于图像令牌的动态视觉转换器 (DiT)，以适应对象尺度和视觉识别的变化，并通过选择多路径特征传播来精心调整图像表达的对象尺度和视觉识别的影响，从而实现更好的性能和良好的复杂度 / 准确度平衡。