Low-Rank Adaptations (LoRAs) have emerged as a powerful and popular technique
in the field of image generation, offering a highly effective way to adapt and
refine pre-trained deep learning models for specific tasks without the need for
comprehensive retraining. By employing pre-trained LoRA models, such as those
representing a specific cat and a particular dog, the objective is to generate
an image that faithfully embodies both animals as defined by the LoRAs.
However, the task of seamlessly blending multiple concept LoRAs to capture a
variety of concepts in one image proves to be a significant challenge. Common
approaches often fall short, primarily because the attention mechanisms within
different LoRA models overlap, leading to scenarios where one concept may be
completely ignored (e.g., omitting the dog) or where concepts are incorrectly
combined (e.g., producing an image of two cats instead of one cat and one dog).
To overcome these issues, CLoRA addresses them by updating the attention maps
of multiple LoRA models and leveraging them to create semantic masks that
facilitate the fusion of latent representations. Our method enables the
creation of composite images that truly reflect the characteristics of each
LoRA, successfully merging multiple concepts or styles. Our comprehensive
evaluations, both qualitative and quantitative, demonstrate that our approach
outperforms existing methodologies, marking a significant advancement in the
field of image generation with LoRAs. Furthermore, we share our source code,
benchmark dataset, and trained LoRA models to promote further research on this
topic.

通过更新多个 LoRA 模型的注意力图并利用它们创建语义掩码来解决在图像生成中多个概念的平滑融合问题，并成功地合并多个概念或样式。

CLoRA：一种对比的方法来组合多个 LoRA 模型

CLoRA: A Contrastive Approach to Compose Multiple LoRA Models

Semantic image synthesis (SIS) aims to generate realistic images that match
given semantic masks. Despite recent advances allowing high-quality results and
precise spatial control, they require a massive semantic segmentation dataset
for training the models. Instead, we propose to employ a pre-trained
unconditional generator and rearrange its feature maps according to proxy
masks. The proxy masks are prepared from the feature maps of random samples in
the generator by simple clustering. The feature rearranger learns to rearrange
original feature maps to match the shape of the proxy masks that are either
from the original sample itself or from random samples. Then we introduce a
semantic mapper that produces the proxy masks from various input conditions
including semantic masks. Our method is versatile across various applications
such as free-form spatial editing of real images, sketch-to-photo, and even
scribble-to-photo. Experiments validate advantages of our method on a range of
datasets: human faces, animal faces, and buildings.

通过使用预训练的无条件生成器和重新排列特征图来生成与给定语义掩模相匹配的逼真图像。该方法在各种应用中表现出优势，包括真实图像的自由空间编辑，素描转照片，甚至涂鸦转照片。

无条件生成器的语义图像合成

Semantic Image Synthesis with Unconditional Generator

Semantically guided conditional Generative Adversarial Networks (cGANs) have
become a popular approach for face editing in recent years. However, most
existing methods introduce semantic masks as direct conditional inputs to the
generator and often require the target masks to perform the corresponding
translation in the RGB space. We propose SeCGAN, a novel label-guided cGAN for
editing face images utilising semantic information without the need to specify
target semantic masks. During training, SeCGAN has two branches of generators
and discriminators operating in parallel, with one trained to translate RGB
images and the other for semantic masks. To bridge the two branches in a
mutually beneficial manner, we introduce a semantic consistency loss which
constrains both branches to have consistent semantic outputs. Whilst both
branches are required during training, the RGB branch is our primary network
and the semantic branch is not needed for inference. Our results on CelebA and
CelebA-HQ demonstrate that our approach is able to generate facial images with
more accurate attributes, outperforming competitive baselines in terms of
Target Attribute Recognition Rate whilst maintaining quality metrics such as
self-supervised Fr\'{e}chet Inception Distance and Inception Score.

通过提出一种语义信息导向的生成对抗网络 SeCGAN，可无需指定目标语义掩码的情况下，利用语义信息进行人脸图像编辑，从而在保持质量指标的前提下，在 CelebA 和 CelebA-HQ 上生成具有更准确属性的面部图像，优于竞争基线项。

SeCGAN：基于语义一致性的平行条件生成式对抗网络用于人脸编辑

SeCGAN: Parallel Conditional Generative Adversarial Networks for Face  Editing via Semantic Consistency

Facial image manipulation has achieved great progress in recent years.
However, previous methods either operate on a predefined set of face attributes
or leave users little freedom to interactively manipulate images. To overcome
these drawbacks, we propose a novel framework termed MaskGAN, enabling diverse
and interactive face manipulation. Our key insight is that semantic masks serve
as a suitable intermediate representation for flexible face manipulation with
fidelity preservation. MaskGAN has two main components: 1) Dense Mapping
Network (DMN) and 2) Editing Behavior Simulated Training (EBST). Specifically,
DMN learns style mapping between a free-form user modified mask and a target
image, enabling diverse generation results. EBST models the user editing
behavior on the source mask, making the overall framework more robust to
various manipulated inputs. Specifically, it introduces dual-editing
consistency as the auxiliary supervision signal. To facilitate extensive
studies, we construct a large-scale high-resolution face dataset with
fine-grained mask annotations named CelebAMask-HQ. MaskGAN is comprehensively
evaluated on two challenging tasks: attribute transfer and style copy,
demonstrating superior performance over other state-of-the-art methods. The
code, models, and dataset are available at
this https URL

提出了一种名为 MaskGAN 的新框架，采用语义化掩模作为中间表示，实现多样化和交互式面部操纵，该框架在两个具有挑战性的任务上进行了综合评估：属性传输和样式复制，在实验证明了优于其他最先进的方法。