Recent years have witnessed the strong power of large text-to-image diffusion
models for the impressive generative capability to create high-fidelity images.
However, it is very tricky to generate desired images using only text prompt as
it often involves complex prompt engineering. An alternative to text prompt is
image prompt, as the saying goes: "an image is worth a thousand words".
Although existing methods of direct fine-tuning from pretrained models are
effective, they require large computing resources and are not compatible with
other base models, text prompt, and structural controls. In this paper, we
present IP-Adapter, an effective and lightweight adapter to achieve image
prompt capability for the pretrained text-to-image diffusion models. The key
design of our IP-Adapter is decoupled cross-attention mechanism that separates
cross-attention layers for text features and image features. Despite the
simplicity of our method, an IP-Adapter with only 22M parameters can achieve
comparable or even better performance to a fully fine-tuned image prompt model.
As we freeze the pretrained diffusion model, the proposed IP-Adapter can be
generalized not only to other custom models fine-tuned from the same base
model, but also to controllable generation using existing controllable tools.
With the benefit of the decoupled cross-attention strategy, the image prompt
can also work well with the text prompt to achieve multimodal image generation.
The project page is available at https://ip-adapter.github.io.

IP-Adapter 是一种有效且轻量级的适配器，通过分离交叉注意力层实现了预训练文本到图像扩散模型的图像提示能力，具有与完全微调的图像提示模型相媲美甚至更好的性能。利用分离的交叉注意力策略，图像提示可以与文本提示良好协作，实现多模态图像生成。

IP-Adapter：面向文本图像扩散模型的文本兼容图像提示适配器

IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image  Diffusion Models

Recent advancements in text-to-image generation with diffusion models have
yielded remarkable results synthesizing highly realistic and diverse images.
However, these models still encounter difficulties when generating images from
prompts that demand spatial or common sense reasoning. We propose to equip
diffusion models with enhanced reasoning capabilities by using off-the-shelf
pretrained large language models (LLMs) in a novel two-stage generation
process. First, we adapt an LLM to be a text-guided layout generator through
in-context learning. When provided with an image prompt, an LLM outputs a scene
layout in the form of bounding boxes along with corresponding individual
descriptions. Second, we steer a diffusion model with a novel controller to
generate images conditioned on the layout. Both stages utilize frozen
pretrained models without any LLM or diffusion model parameter optimization. We
validate the superiority of our design by demonstrating its ability to
outperform the base diffusion model in accurately generating images according
to prompts that necessitate both language and spatial reasoning. Additionally,
our method naturally allows dialog-based scene specification and is able to
handle prompts in a language that is not well-supported by the underlying
diffusion model.

该论文介绍了一种使用预训练大型语言模型对扩散模型进行增强推理能力的方法，包括两个阶段：第一阶段使用大型语言模型生成场景布局；第二阶段使用一个新的控制器，来生成与布局条件相符的图像。实验结果表明，该方法能更准确地生成需要语言和空间推理的图像。