We consider the problem of customizing text-to-image diffusion models with
user-supplied reference images. Given new prompts, the existing methods can
capture the key concept from the reference images but fail to align the
generated image with the prompt. In this work, we seek to address this key
issue by proposing new methods that can easily be used in conjunction with
existing customization methods that optimize the embeddings/weights at various
intermediate stages of the text encoding process.
The first contribution of this paper is a dissection of the various stages of
the text encoding process leading up to the conditioning vector for
text-to-image models. We take a holistic view of existing customization methods
and notice that key and value outputs from this process differs substantially
from their corresponding baseline (non-customized) models (e.g., baseline
stable diffusion). While this difference does not impact the concept being
customized, it leads to other parts of the generated image not being aligned
with the prompt (see first row in Fig 1). Further, we also observe that these
keys and values allow independent control various aspects of the final
generation, enabling semantic manipulation of the output. Taken together, the
features spanning these keys and values, serve as the basis for our next
contribution where we fix the aforementioned issues with existing methods. We
propose a new post-processing algorithm, \textbf{AlignIT}, that infuses the
keys and values for the concept of interest while ensuring the keys and values
for all other tokens in the input prompt are unchanged.
Our proposed method can be plugged in directly to existing customization
methods, leading to a substantial performance improvement in the alignment of
the final result with the input prompt while retaining the customization
quality.

我们提出了一种新的后处理算法 AlignIT，该算法将感兴趣的概念的键和值与输入提示中的所有其他标记的键和值保持不变，从而解决了现有方法存在的问题，并显著提高了与输入提示的对齐性能。