In text-to-image personalization, a timely and crucial challenge is the
tendency of generated images overfitting to the biases present in the reference
images. We initiate our study with a comprehensive categorization of the biases
into background, nearby-object, tied-object, substance (in style
re-contextualization), and pose biases. These biases manifest in the generated
images due to their entanglement into the subject embedding. This undesired
embedding entanglement not only results in the reflection of biases from the
reference images into the generated images but also notably diminishes the
alignment of the generated images with the given generation prompt. To address
this challenge, we propose SID~(Selectively Informative Description), a text
description strategy that deviates from the prevalent approach of only
characterizing the subject's class identification. SID is generated utilizing
multimodal GPT-4 and can be seamlessly integrated into optimization-based
models. We present comprehensive experimental results along with analyses of
cross-attention maps, subject-alignment, non-subject-disentanglement, and
text-alignment.

针对生成的图像在参考图像中出现的偏见问题，本研究提出了 SEB（选择性信息化描述），它是一种与现有方法不同的文本描述策略。通过利用多模态 GPT-4 生成 SEB，可以无缝集成到基于优化的模型中，解决了生成图像与给定生成提示不对齐的问题。

有选择性的信息描述可以减少文本到图像个性化中的不必要嵌入纠缠

Selectively Informative Description can Reduce Undesired Embedding  Entanglements in Text-to-Image Personalization

The objective of text-to-image (T2I) personalization is to customize a
diffusion model to a user-provided reference concept, generating diverse images
of the concept aligned with the target prompts. Conventional methods
representing the reference concepts using unique text embeddings often fail to
accurately mimic the appearance of the reference. To address this, one solution
may be explicitly conditioning the reference images into the target denoising
process, known as key-value replacement. However, prior works are constrained
to local editing since they disrupt the structure path of the pre-trained T2I
model. To overcome this, we propose a novel plug-in method, called
DreamMatcher, which reformulates T2I personalization as semantic matching.
Specifically, DreamMatcher replaces the target values with reference values
aligned by semantic matching, while leaving the structure path unchanged to
preserve the versatile capability of pre-trained T2I models for generating
diverse structures. We also introduce a semantic-consistent masking strategy to
isolate the personalized concept from irrelevant regions introduced by the
target prompts. Compatible with existing T2I models, DreamMatcher shows
significant improvements in complex scenarios. Intensive analyses demonstrate
the effectiveness of our approach.

T2I 个性化通过语义匹配替换目标值与参考值，将 T2I 个性化重新构想为语义匹配，避免破坏预训练模型的多样性能力，并引入了一种语义一致的遮罩策略以隔离个性化概念与目标提示之外的无关区域。

DreamMatcher： 外貌匹配自注意力用于语义一致的文本到图像个性化

DreamMatcher: Appearance Matching Self-Attention for  Semantically-Consistent Text-to-Image Personalization

Text-to-image (T2I) personalization allows users to guide the creative image
generation process by combining their own visual concepts in natural language
prompts. Recently, encoder-based techniques have emerged as a new effective
approach for T2I personalization, reducing the need for multiple images and
long training times. However, most existing encoders are limited to a
single-class domain, which hinders their ability to handle diverse concepts. In
this work, we propose a domain-agnostic method that does not require any
specialized dataset or prior information about the personalized concepts. We
introduce a novel contrastive-based regularization technique to maintain high
fidelity to the target concept characteristics while keeping the predicted
embeddings close to editable regions of the latent space, by pushing the
predicted tokens toward their nearest existing CLIP tokens. Our experimental
results demonstrate the effectiveness of our approach and show how the learned
tokens are more semantic than tokens predicted by unregularized models. This
leads to a better representation that achieves state-of-the-art performance
while being more flexible than previous methods.

本文提出了一种基于编码器的文本到图像（T2I）个性化生成方法，使用一种对领域不敏感的对比度正则化技术来实现目标概念特征的高保真度，并将预测的嵌入值保持在可编辑的潜在空间区域，从而实现了基于 CLIP（一种联合学习图像和文本的新型表示方法）标记的最短距离。实验结果证明了我们方法的有效性和学习标记比未规范化模型预测的标记更具语义性的优点，并且比以前的方法更加灵活，表现出最先进的性能。