Recent advancements in diffusion models have notably improved the perceptual
quality of generated images in text-to-image synthesis tasks. However,
diffusion models often struggle to produce images that accurately reflect the
intended semantics of the associated text prompts. We examine cross-attention
layers in diffusion models and observe a propensity for these layers to
disproportionately focus on certain tokens during the generation process,
thereby undermining semantic fidelity. To address the issue of dominant
attention, we introduce attention regulation, a computation-efficient
on-the-fly optimization approach at inference time to align attention maps with
the input text prompt. Notably, our method requires no additional training or
fine-tuning and serves as a plug-in module on a model. Hence, the generation
capacity of the original model is fully preserved. We compare our approach with
alternative approaches across various datasets, evaluation metrics, and
diffusion models. Experiment results show that our method consistently
outperforms other baselines, yielding images that more faithfully reflect the
desired concepts with reduced computation overhead. Code is available at
this https URL

我们提出了一种注意力调节方法，通过实时优化来对齐注意力图与输入文本提示，从而解决扩散模型中对某些令牌过分关注而导致语义逼真度下降的问题。在各种数据集、评估指标和扩散模型上进行了实验比较，结果表明我们的方法在减少计算开销的同时，始终优于其他基准方法，生成的图像更加忠实地反映了所需的概念。