Recent advancements in controllable human image generation have led to
zero-shot generation using structural signals (e.g., pose, depth) or facial
appearance. Yet, generating human images conditioned on multiple parts of human
appearance remains challenging. Addressing this, we introduce Parts2Whole, a
novel framework designed for generating customized portraits from multiple
reference images, including pose images and various aspects of human
appearance. To achieve this, we first develop a semantic-aware appearance
encoder to retain details of different human parts, which processes each image
based on its textual label to a series of multi-scale feature maps rather than
one image token, preserving the image dimension. Second, our framework supports
multi-image conditioned generation through a shared self-attention mechanism
that operates across reference and target features during the diffusion
process. We enhance the vanilla attention mechanism by incorporating mask
information from the reference human images, allowing for the precise selection
of any part. Extensive experiments demonstrate the superiority of our approach
over existing alternatives, offering advanced capabilities for multi-part
controllable human image customization. See our project page at
this https URL

通过多个参考图像生成定制肖像的新框架 Parts2Whole，利用语义感知的外观编码器和共享自注意力机制，在保留细节的同时支持多图像条件生成，实现了对多部分可控人体图像的高级定制能力。

从部件到整体：可控人体图像生成的统一参考框架

From Parts to Whole: A Unified Reference Framework for Controllable  Human Image Generation

Controllable human image generation (HIG) has numerous real-life
applications. State-of-the-art solutions, such as ControlNet and T2I-Adapter,
introduce an additional learnable branch on top of the frozen pre-trained
stable diffusion (SD) model, which can enforce various conditions, including
skeleton guidance of HIG. While such a plug-and-play approach is appealing, the
inevitable and uncertain conflicts between the original images produced from
the frozen SD branch and the given condition incur significant challenges for
the learnable branch, which essentially conducts image feature editing for
condition enforcement. In this work, we propose a native skeleton-guided
diffusion model for controllable HIG called HumanSD. Instead of performing
image editing with dual-branch diffusion, we fine-tune the original SD model
using a novel heatmap-guided denoising loss. This strategy effectively and
efficiently strengthens the given skeleton condition during model training
while mitigating the catastrophic forgetting effects. HumanSD is fine-tuned on
the assembly of three large-scale human-centric datasets with text-image-pose
information, two of which are established in this work. As shown in Figure 1,
HumanSD outperforms ControlNet in terms of accurate pose control and image
quality, particularly when the given skeleton guidance is sophisticated.

本文提出了一种基于本地细骨架引导扩散模型的可控人类图像生成方法，称为 HumanSD，该方法使用新颖的热图引导去噪损失来微调原始稳定扩散模型，从而有效地增强了训练模型时给定的骨骼条件，并提高了图像质量。