We introduce X-Adapter, a universal upgrader to enable the pretrained
plug-and-play modules (e.g., ControlNet, LoRA) to work directly with the
upgraded text-to-image diffusion model (e.g., SDXL) without further retraining.
We achieve this goal by training an additional network to control the frozen
upgraded model with the new text-image data pairs. In detail, X-Adapter keeps a
frozen copy of the old model to preserve the connectors of different plugins.
Additionally, X-Adapter adds trainable mapping layers that bridge the decoders
from models of different versions for feature remapping. The remapped features
will be used as guidance for the upgraded model. To enhance the guidance
ability of X-Adapter, we employ a null-text training strategy for the upgraded
model. After training, we also introduce a two-stage denoising strategy to
align the initial latents of X-Adapter and the upgraded model. Thanks to our
strategies, X-Adapter demonstrates universal compatibility with various plugins
and also enables plugins of different versions to work together, thereby
expanding the functionalities of diffusion community. To verify the
effectiveness of the proposed method, we conduct extensive experiments and the
results show that X-Adapter may facilitate wider application in the upgraded
foundational diffusion model.

通过使用 X-Adapter，我们成功实现了升级后的文本到图像扩散模型（如 SDXL）与预训练的即插即用模块（如 ControlNet、LoRA）直接配合工作，无需进一步的重新训练。我们通过训练额外的网络来控制冻结的升级模型，使用新的文本 - 图像数据对进行指导，同时保留不同版本插件的连接器，并通过可训练的映射层将不同版本模型的解码器连接起来进行特征映射。X-Adapter 通过使用空文本训练策略和两阶段去噪策略提高了其指导能力，并在实验证明其与各种插件具有通用兼容性，并使不同版本的插件能够共同工作，从而扩展了扩散社区的功能。我们通过广泛的实验验证了所提出方法的有效性，结果表明 X-Adapter 在升级的基础扩散模型中可能有更广泛的应用。