Recent advances in text-to-image diffusion models have enabled the
photorealistic generation of images from text prompts. Despite the great
progress, existing models still struggle to generate compositional
multi-concept images naturally, limiting their ability to visualize human
imagination. While several recent works have attempted to address this issue,
they either introduce additional training or adopt guidance at inference time.
In this work, we consider a more ambitious goal: natural multi-concept
generation using a pre-trained diffusion model, and with almost no extra cost.
To achieve this goal, we identify the limitations in the text embeddings used
for the pre-trained text-to-image diffusion models. Specifically, we observe
concept dominance and non-localized contribution that severely degrade
multi-concept generation performance. We further design a minimal low-cost
solution that overcomes the above issues by tweaking (not re-training) the text
embeddings for more realistic multi-concept text-to-image generation. Our
Correction by Similarities method tweaks the embedding of concepts by
collecting semantic features from most similar tokens to localize the
contribution. To avoid mixing features of concepts, we also apply Cross-Token
Non-Maximum Suppression, which excludes the overlap of contributions from
different concepts. Experiments show that our approach outperforms previous
methods in text-to-image, image manipulation, and personalization tasks,
despite not introducing additional training or inference costs to the diffusion
steps.

通过对预训练文本到图像扩散模型中的文本嵌入进行微调，我们设计了一种低成本的解决方案，实现自然多概念文本到图像生成，并在扩散步骤中不增加附加的训练或推理成本。我们的方法通过收集与最相似的令牌的语义特征来定位贡献，并应用交叉令牌非极大值抑制来避免不同概念之间的特征混合，从而在文本到图像、图像操作和个性化任务中优于以前的方法。