Diffusion models are widely recognized for generating high-quality and
diverse images, but their poor real-time performance has led to numerous
acceleration works, primarily focusing on UNet-based structures. With the more
successful results achieved by diffusion transformers (DiT), there is still a
lack of exploration regarding the impact of DiT structure on generation, as
well as the absence of an acceleration framework tailored to the DiT
architecture. To tackle these challenges, we conduct an investigation into the
correlation between DiT blocks and image generation. Our findings reveal that
the front blocks of DiT are associated with the outline of the generated
images, while the rear blocks are linked to the details. Based on this insight,
we propose an overall training-free inference acceleration framework
$\Delta$-DiT: using a designed cache mechanism to accelerate the rear DiT
blocks in the early sampling stages and the front DiT blocks in the later
stages. Specifically, a DiT-specific cache mechanism called $\Delta$-Cache is
proposed, which considers the inputs of the previous sampling image and reduces
the bias in the inference. Extensive experiments on PIXART-$\alpha$ and DiT-XL
demonstrate that the $\Delta$-DiT can achieve a $1.6\times$ speedup on the
20-step generation and even improves performance in most cases. In the scenario
of 4-step consistent model generation and the more challenging $1.12\times$
acceleration, our method significantly outperforms existing methods. Our code
will be publicly available.

通过调查 DiT 块与图像生成之间的相关性，发现 DiT 的前述块与生成图像的轮廓有关，而后述块与细节有关。基于此，我们提出了一个全面的无需训练的推断加速框架 Delta-DiT，采用设计的缓存机制来加速早期采样阶段的后述 DiT 块和后期阶段的前述 DiT 块。在 PIXART-alpha 和 DiT-XL 上的大量实验证明，Delta-DiT 在 20 步生成时可以实现 1.6 倍加速，并在大多数情况下性能提高。在 4 步一致模型生成和更具挑战性的 1.12 倍加速的情况下，我们的方法显著优于现有方法。我们的代码将公开提供。