Public large-scale text-to-image diffusion models, such as Stable Diffusion, have gained significant attention from the community. These models can be easily customized for new concepts using low-rank adaptations (LoRAs). However, the utilization of multiple concept LoRAs to jointly support multiple customized concepts presents a challenge. We refer to this scenario as decentralized multi-concept customization, which involves single-client concept tuning and center-node concept fusion. In this paper, we propose a new framework called Mix-of-Show that addresses the challenges of decentralized multi-concept customization, including concept conflicts resulting from existing single-client LoRA tuning and identity loss during model fusion. Mix-of-Show adopts an embedding-decomposed LoRA (ED-LoRA) for single-client tuning and gradient fusion for the center node to preserve the in-domain essence of single concepts and support theoretically limitless concept fusion. Additionally, we introduce regionally controllable sampling, which extends spatially controllable sampling (e.g., ControlNet and T2I-Adaptor) to address attribute binding and missing object problems in multi-concept sampling. Extensive experiments demonstrate that Mix-of-Show is capable of composing multiple customized concepts with high fidelity, including characters, objects, and scenes.

本文提出了一种名为Mix-of-Show的新框架，它采用嵌入分解的LoRA (ED-LoRA)和梯度融合来处理多客户端单一概念调整和中心节点概念融合中的概念冲突和身份丢失问题，同时引入区域可控采样以扩展空间可控采样以解决多概念抽样中的属性绑定和缺失物体问题。实验证明Mix-of-Show能够高度还原多种自定义概念，包括文字、物体和场景。

Mix-of-Show: 为扩散模型的多概念定制提供低秩自适应去中心化方案