Low-Rank Adaptations (LoRAs) have emerged as a powerful and popular technique
in the field of image generation, offering a highly effective way to adapt and
refine pre-trained deep learning models for specific tasks without the need for
comprehensive retraining. By employing pre-trained LoRA models, such as those
representing a specific cat and a particular dog, the objective is to generate
an image that faithfully embodies both animals as defined by the LoRAs.
However, the task of seamlessly blending multiple concept LoRAs to capture a
variety of concepts in one image proves to be a significant challenge. Common
approaches often fall short, primarily because the attention mechanisms within
different LoRA models overlap, leading to scenarios where one concept may be
completely ignored (e.g., omitting the dog) or where concepts are incorrectly
combined (e.g., producing an image of two cats instead of one cat and one dog).
To overcome these issues, CLoRA addresses them by updating the attention maps
of multiple LoRA models and leveraging them to create semantic masks that
facilitate the fusion of latent representations. Our method enables the
creation of composite images that truly reflect the characteristics of each
LoRA, successfully merging multiple concepts or styles. Our comprehensive
evaluations, both qualitative and quantitative, demonstrate that our approach
outperforms existing methodologies, marking a significant advancement in the
field of image generation with LoRAs. Furthermore, we share our source code,
benchmark dataset, and trained LoRA models to promote further research on this
topic.

通过更新多个 LoRA 模型的注意力图并利用它们创建语义掩码来解决在图像生成中多个概念的平滑融合问题，并成功地合并多个概念或样式。

CLoRA：一种对比的方法来组合多个 LoRA 模型

CLoRA: A Contrastive Approach to Compose Multiple LoRA Models

Methods for finetuning generative models for concept-driven personalization
generally achieve strong results for subject-driven or style-driven generation.
Recently, low-rank adaptations (LoRA) have been proposed as a
parameter-efficient way of achieving concept-driven personalization. While
recent work explores the combination of separate LoRAs to achieve joint
generation of learned styles and subjects, existing techniques do not reliably
address the problem; they often compromise either subject fidelity or style
fidelity. We propose ZipLoRA, a method to cheaply and effectively merge
independently trained style and subject LoRAs in order to achieve generation of
any user-provided subject in any user-provided style. Experiments on a wide
range of subject and style combinations show that ZipLoRA can generate
compelling results with meaningful improvements over baselines in subject and
style fidelity while preserving the ability to recontextualize. Project page:
this https URL

通过低秩适配（LoRA）方法，提出了 ZipLoRA，一种能够高效且在主题和风格保真度方面有显著改进的独立训练风格和主题 LoRA 的合并方法。

ZipLoRA: 通过有效地合并 LoRA 以任意主题任意风格

ZipLoRA: Any Subject in Any Style by Effectively Merging LoRAs

Low-rank adaptations (LoRA) are often employed to fine-tune large language
models (LLMs) for new tasks. This paper investigates LoRA composability for
cross-task generalization and introduces LoraHub, a strategic framework devised
for the purposive assembly of LoRA modules trained on diverse given tasks, with
the objective of achieving adaptable performance on unseen tasks. With just a
few examples from a novel task, LoraHub enables the fluid combination of
multiple LoRA modules, eradicating the need for human expertise. Notably, the
composition requires neither additional model parameters nor gradients. Our
empirical results, derived from the Big-Bench Hard (BBH) benchmark, suggest
that LoraHub can effectively mimic the performance of in-context learning in
few-shot scenarios, excluding the necessity of in-context examples alongside
each inference input. A significant contribution of our research is the
fostering of a community for LoRA, where users can share their trained LoRA
modules, thereby facilitating their application to new tasks. We anticipate
this resource will widen access to and spur advancements in general
intelligence as well as LLMs in production. Code will be available at
this https URL

LoRAHub 是一个用于组合多个训练在不同任务上的 LoRA 模块的战略性框架，旨在实现在未知任务上的适应性性能，可以有效地模拟在少样本情况下的上下文学习表现，无需上下文示例。