Diffusion models are widely recognized for generating high-quality and
diverse images, but their poor real-time performance has led to numerous
acceleration works, primarily focusing on UNet-based structures. With the more
successful results achieved by diffusion transformers (DiT), there is still a
lack of exploration regarding the impact of DiT structure on generation, as
well as the absence of an acceleration framework tailored to the DiT
architecture. To tackle these challenges, we conduct an investigation into the
correlation between DiT blocks and image generation. Our findings reveal that
the front blocks of DiT are associated with the outline of the generated
images, while the rear blocks are linked to the details. Based on this insight,
we propose an overall training-free inference acceleration framework
$\Delta$-DiT: using a designed cache mechanism to accelerate the rear DiT
blocks in the early sampling stages and the front DiT blocks in the later
stages. Specifically, a DiT-specific cache mechanism called $\Delta$-Cache is
proposed, which considers the inputs of the previous sampling image and reduces
the bias in the inference. Extensive experiments on PIXART-$\alpha$ and DiT-XL
demonstrate that the $\Delta$-DiT can achieve a $1.6\times$ speedup on the
20-step generation and even improves performance in most cases. In the scenario
of 4-step consistent model generation and the more challenging $1.12\times$
acceleration, our method significantly outperforms existing methods. Our code
will be publicly available.

通过调查 DiT 块与图像生成之间的相关性，发现 DiT 的前述块与生成图像的轮廓有关，而后述块与细节有关。基于此，我们提出了一个全面的无需训练的推断加速框架 Delta-DiT，采用设计的缓存机制来加速早期采样阶段的后述 DiT 块和后期阶段的前述 DiT 块。在 PIXART-alpha 和 DiT-XL 上的大量实验证明，Delta-DiT 在 20 步生成时可以实现 1.6 倍加速，并在大多数情况下性能提高。在 4 步一致模型生成和更具挑战性的 1.12 倍加速的情况下，我们的方法显著优于现有方法。我们的代码将公开提供。

$Δ$-DiT：一种为扩散变换器量身定制的无需训练的加速方法

$Δ$-DiT: A Training-Free Acceleration Method Tailored for Diffusion  Transformers

Diffusion models have shown remarkable performance in image generation in
recent years. However, due to a quadratic increase in memory during generating
ultra-high-resolution images (e.g. 4096*4096), the resolution of generated
images is often limited to 1024*1024. In this work. we propose a unidirectional
block attention mechanism that can adaptively adjust the memory overhead during
the inference process and handle global dependencies. Building on this module,
we adopt the DiT structure for upsampling and develop an infinite
super-resolution model capable of upsampling images of various shapes and
resolutions. Comprehensive experiments show that our model achieves SOTA
performance in generating ultra-high-resolution images in both machine and
human evaluation. Compared to commonly used UNet structures, our model can save
more than 5x memory when generating 4096*4096 images. The project URL is
this https URL

通过引入单向注意力机制和 DiT 结构，我们提出了一种自适应调整推理过程中的内存负荷和处理全局依赖性的无限超分模型，实验结果表明，在生成超高分辨率图像方面，我们的模型在机器和人类评估方面都达到了最先进的性能。与常用的 UNet 结构相比，在生成 4096*4096 图像时，我们的模型可以节省超过 5 倍的内存。