Recent studies have demonstrated that diffusion models are capable of
generating high-quality samples, but their quality heavily depends on sampling
guidance techniques, such as classifier guidance (CG) and classifier-free
guidance (CFG). These techniques are often not applicable in unconditional
generation or in various downstream tasks such as image restoration. In this
paper, we propose a novel sampling guidance, called Perturbed-Attention
Guidance (PAG), which improves diffusion sample quality across both
unconditional and conditional settings, achieving this without requiring
additional training or the integration of external modules. PAG is designed to
progressively enhance the structure of samples throughout the denoising
process. It involves generating intermediate samples with degraded structure by
substituting selected self-attention maps in diffusion U-Net with an identity
matrix, by considering the self-attention mechanisms' ability to capture
structural information, and guiding the denoising process away from these
degraded samples. In both ADM and Stable Diffusion, PAG surprisingly improves
sample quality in conditional and even unconditional scenarios. Moreover, PAG
significantly improves the baseline performance in various downstream tasks
where existing guidances such as CG or CFG cannot be fully utilized, including
ControlNet with empty prompts and image restoration such as inpainting and
deblurring.

基于扩散模型的采样指导技术在无条件生成和各种下游任务中限制较多，本研究提出一种称为 Perturbed-Attention Guidance (PAG) 的新型采样指导方法，通过渐进式增强样本结构，改善扩散采样的质量，并在条件和无条件场景下实现了显著性提升，同时在空提示的 ControlNet 以及图像修复（如修补和去模糊）等各种下游任务中也取得了显著的性能提升。