We introduce X-Adapter, a universal upgrader to enable the pretrained
plug-and-play modules (e.g., ControlNet, LoRA) to work directly with the
upgraded text-to-image diffusion model (e.g., SDXL) without further retraining.
We achieve this goal by training an additional network to control the frozen
upgraded model with the new text-image data pairs. In detail, X-Adapter keeps a
frozen copy of the old model to preserve the connectors of different plugins.
Additionally, X-Adapter adds trainable mapping layers that bridge the decoders
from models of different versions for feature remapping. The remapped features
will be used as guidance for the upgraded model. To enhance the guidance
ability of X-Adapter, we employ a null-text training strategy for the upgraded
model. After training, we also introduce a two-stage denoising strategy to
align the initial latents of X-Adapter and the upgraded model. Thanks to our
strategies, X-Adapter demonstrates universal compatibility with various plugins
and also enables plugins of different versions to work together, thereby
expanding the functionalities of diffusion community. To verify the
effectiveness of the proposed method, we conduct extensive experiments and the
results show that X-Adapter may facilitate wider application in the upgraded
foundational diffusion model.

通过使用 X-Adapter，我们成功实现了升级后的文本到图像扩散模型（如 SDXL）与预训练的即插即用模块（如 ControlNet、LoRA）直接配合工作，无需进一步的重新训练。我们通过训练额外的网络来控制冻结的升级模型，使用新的文本 - 图像数据对进行指导，同时保留不同版本插件的连接器，并通过可训练的映射层将不同版本模型的解码器连接起来进行特征映射。X-Adapter 通过使用空文本训练策略和两阶段去噪策略提高了其指导能力，并在实验证明其与各种插件具有通用兼容性，并使不同版本的插件能够共同工作，从而扩展了扩散社区的功能。我们通过广泛的实验验证了所提出方法的有效性，结果表明 X-Adapter 在升级的基础扩散模型中可能有更广泛的应用。

X-Adapter：增加插件对升级扩散模型的通用兼容性

X-Adapter: Adding Universal Compatibility of Plugins for Upgraded  Diffusion Model

Humans learn language via multi-modal knowledge. However, due to the
text-only pre-training scheme, most existing pre-trained language models (PLMs)
are hindered from the multi-modal information.
To inject visual knowledge into PLMs, existing methods incorporate either the
text or image encoder of vision-language models (VLMs) to encode the visual
information and update all the original parameters of PLMs for knowledge
fusion.
In this paper, we propose a new plug-and-play module, X-adapter, to flexibly
leverage the aligned visual and textual knowledge learned in pre-trained VLMs
and efficiently inject them into PLMs.
Specifically, we insert X-adapters into PLMs, and only the added parameters
are updated during adaptation.
To fully exploit the potential in VLMs, X-adapters consist of two
sub-modules, V-expert and T-expert, to fuse VLMs' image and text
representations, respectively.
We can opt for activating different sub-modules depending on the downstream
tasks.
Experimental results show that our method can significantly improve the
performance on object-color reasoning and natural language understanding (NLU)
tasks compared with PLM baselines.

本文提出了一种新的插入式模块 X-adapter，用于将预训练的 VLMs 的对齐视觉和文本知识灵活地融入 PLMs 中，以提高对象 - 颜色推理和自然语言理解 (NLU) 任务性能。