Recent developments in large-scale pre-trained text-to-image diffusion models
have significantly improved the generation of high-fidelity images,
particularly with the emergence of diffusion models based on transformer
architecture (DiTs). Among these diffusion models, diffusion transformers have
demonstrated superior image generation capabilities, boosting lower FID scores
and higher scalability. However, deploying large-scale DiT models can be
expensive due to their extensive parameter numbers. Although existing research
has explored efficient deployment techniques for diffusion models such as model
quantization, there is still little work concerning DiT-based models. To tackle
this research gap, in this paper, we propose TerDiT, a quantization-aware
training (QAT) and efficient deployment scheme for ternary diffusion models
with transformers. We focus on the ternarization of DiT networks and scale
model sizes from 600M to 4.2B. Our work contributes to the exploration of
efficient deployment strategies for large-scale DiT models, demonstrating the
feasibility of training extremely low-bit diffusion transformer models from
scratch while maintaining competitive image generation capacities compared to
full-precision models. Code will be available at
this https URL

我们提出了 TerDiT，一种针对具有 transformer 的三态扩散模型的量化感知训练（QAT）和有效部署方案，致力于探索大规模 DiT 模型的高效部署策略，展示了从头开始训练极低比特扩散变压器模型的可行性，同时保持与全精度模型相比有竞争力的图像生成能力。