The remarkable advancement in text-to-image generation models significantly
boosts the research in ID customization generation. However, existing
personalization methods cannot simultaneously satisfy high fidelity and
high-efficiency requirements. Their main bottleneck lies in the prompt image
encoder, which produces weak alignment signals with the text-to-image model and
significantly increased model size. Towards this end, we propose a lightweight
Inv-Adapter, which first extracts diffusion-domain representations of ID images
utilizing a pre-trained text-to-image model via DDIM image inversion, without
additional image encoder. Benefiting from the high alignment of the extracted
ID prompt features and the intermediate features of the text-to-image model, we
then embed them efficiently into the base text-to-image model by carefully
designing a lightweight attention adapter. We conduct extensive experiments to
assess ID fidelity, generation loyalty, speed, and training parameters, all of
which show that the proposed Inv-Adapter is highly competitive in ID
customization generation and model scale.

利用预训练的文本图像模型通过 DDIM 图像反转提取 ID 图像的传播域表示，不需要额外的图像编码器，然后通过精心设计的轻量级注意力适配器将提取的 ID 提示特征和文本图像模型的中间特征有效地嵌入到基础文本图像模型中，从而为 ID 定制生成和模型规模上提出的 Inv-Adapter 在 ID 忠实度、生成忠诚度、速度和训练参数等方面进行了广泛的实验评估表明其具有很高的竞争力。