Subject-driven text-to-image diffusion models empower users to tailor the
model to new concepts absent in the pre-training dataset using a few sample
images. However, prevalent subject-driven models primarily rely on
single-concept input images, facing challenges in specifying the target concept
when dealing with multi-concept input images. To this end, we introduce a
textual localized text-to-image model (Texual Localization) to handle
multi-concept input images. During fine-tuning, our method incorporates a novel
cross-attention guidance to decompose multiple concepts, establishing distinct
connections between the visual representation of the target concept and the
identifier token in the text prompt. Experimental results reveal that our
method outperforms or performs comparably to the baseline models in terms of
image fidelity and image-text alignment on multi-concept input images. In
comparison to Custom Diffusion, our method with hard guidance achieves CLIP-I
scores that are 7.04%, 8.13% higher and CLIP-T scores that are 2.22%, 5.85%
higher in single-concept and multi-concept generation, respectively. Notably,
our method generates cross-attention maps consistent with the target concept in
the generated images, a capability absent in existing models.

利用主题驱动的文本到图像扩散模型，用户可以根据少量示例图像来定制模型来处理预训练数据集中不存在的新概念。然而，现有的主题驱动模型主要依赖于单一概念的输入图像，在处理多概念输入图像时面临着目标概念的指定困难。为此，我们引入了一种文本本地化的文本到图像模型（Texual Localization）来处理多概念输入图像。在微调过程中，我们的方法采用了一种新颖的交叉注意力引导方法，分解多个概念，在文本提示中建立目标概念的视觉表示与标识符令牌之间的明确连接。实验结果表明，在多概念输入图像上，我们的方法在图像保真度和图像文本对齐方面优于或相当于基准模型。与自定义扩散方法相比，我们的方法通过硬引导实现了单一概念生成的 CLIP-I 得分相对提高 7.04％，8.13％，多概念生成的 CLIP-T 得分相对提高 2.22％，5.85％。值得注意的是，我们的方法能够生成与生成图像中目标概念一致的交叉注意力映射，这是现有模型所没有的能力。