In text-to-image generation tasks, the advancements of diffusion models have
facilitated the fidelity of generated results. However, these models encounter
challenges when processing text prompts containing multiple entities and
attributes. The uneven distribution of attention results in the issues of
entity leakage and attribute misalignment. Training from scratch to address
this issue requires numerous labeled data and is resource-consuming. Motivated
by this, we propose an attribution-focusing mechanism, a training-free
phase-wise mechanism by modulation of attention for diffusion model. One of our
core ideas is to guide the model to concentrate on the corresponding syntactic
components of the prompt at distinct timesteps. To achieve this, we incorporate
a temperature control mechanism within the early phases of the self-attention
modules to mitigate entity leakage issues. An object-focused masking scheme and
a phase-wise dynamic weight control mechanism are integrated into the
cross-attention modules, enabling the model to discern the affiliation of
semantic information between entities more effectively. The experimental
results in various alignment scenarios demonstrate that our model attain better
image-text alignment with minimal additional computational cost.

通过注意力机制的调节，我们提出了一种用于扩散模型的无需训练的逐阶段聚焦机制，旨在解决多实体和属性的文本提示处理中的注意力分布不均问题。我们的实验结果证明，我们的模型在各种情况下都能够以最小的计算成本实现更好的图像与文本的对齐。