Diffusion models with large-scale pre-training have achieved significant
success in the field of visual content generation, particularly exemplified by
Diffusion Transformers (DiT). However, DiT models have faced challenges with
scalability and quadratic complexity efficiency. In this paper, we aim to
leverage the long sequence modeling capability of Gated Linear Attention (GLA)
Transformers, expanding its applicability to diffusion models. We introduce
Diffusion Gated Linear Attention Transformers (DiG), a simple, adoptable
solution with minimal parameter overhead, following the DiT design, but
offering superior efficiency and effectiveness. In addition to better
performance than DiT, DiG-S/2 exhibits $2.5\times$ higher training speed than
DiT-S/2 and saves $75.7\%$ GPU memory at a resolution of $1792 \times 1792$.
Moreover, we analyze the scalability of DiG across a variety of computational
complexity. DiG models, with increased depth/width or augmentation of input
tokens, consistently exhibit decreasing FID. We further compare DiG with other
subquadratic-time diffusion models. With the same model size, DiG-XL/2 is
$4.2\times$ faster than the recent Mamba-based diffusion model at a $1024$
resolution, and is $1.8\times$ faster than DiT with CUDA-optimized
FlashAttention-2 under the $2048$ resolution. All these results demonstrate its
superior efficiency among the latest diffusion models. Code is released at
this https URL

本文基于 Gated Linear Attention Transformers 构建一个可用于扩展预训练扩散模型的简单、易于采用的解决方案，其在效率和效果上优于 Diffusion Transformers 模型。