Recent years have witnessed the strong power of large text-to-image diffusion
models for the impressive generative capability to create high-fidelity images.
However, it is very tricky to generate desired images using only text prompt as
it often involves complex prompt engineering. An alternative to text prompt is
image prompt, as the saying goes: "an image is worth a thousand words".
Although existing methods of direct fine-tuning from pretrained models are
effective, they require large computing resources and are not compatible with
other base models, text prompt, and structural controls. In this paper, we
present IP-Adapter, an effective and lightweight adapter to achieve image
prompt capability for the pretrained text-to-image diffusion models. The key
design of our IP-Adapter is decoupled cross-attention mechanism that separates
cross-attention layers for text features and image features. Despite the
simplicity of our method, an IP-Adapter with only 22M parameters can achieve
comparable or even better performance to a fully fine-tuned image prompt model.
As we freeze the pretrained diffusion model, the proposed IP-Adapter can be
generalized not only to other custom models fine-tuned from the same base
model, but also to controllable generation using existing controllable tools.
With the benefit of the decoupled cross-attention strategy, the image prompt
can also work well with the text prompt to achieve multimodal image generation.
The project page is available at https://ip-adapter.github.io.

IP-Adapter 是一种有效且轻量级的适配器，通过分离交叉注意力层实现了预训练文本到图像扩散模型的图像提示能力，具有与完全微调的图像提示模型相媲美甚至更好的性能。利用分离的交叉注意力策略，图像提示可以与文本提示良好协作，实现多模态图像生成。