Geometry- and appearance-controlled full-body human image generation is an
interesting but challenging task. Existing solutions are either unconditional
or dependent on coarse conditions (e.g., pose, text), thus lacking explicit
geometry and appearance control of body and garment. Sketching offers such
editing ability and has been adopted in various sketch-based face generation
and editing solutions. However, directly adapting sketch-based face generation
to full-body generation often fails to produce high-fidelity and diverse
results due to the high complexity and diversity in the pose, body shape, and
garment shape and texture. Recent geometrically controllable diffusion-based
methods mainly rely on prompts to generate appearance and it is hard to balance
the realism and the faithfulness of their results to the sketch when the input
is coarse. This work presents Sketch2Human, the first system for controllable
full-body human image generation guided by a semantic sketch (for geometry
control) and a reference image (for appearance control). Our solution is based
on the latent space of StyleGAN-Human with inverted geometry and appearance
latent codes as input. Specifically, we present a sketch encoder trained with a
large synthetic dataset sampled from StyleGAN-Human's latent space and directly
supervised by sketches rather than real images. Considering the entangled
information of partial geometry and texture in StyleGAN-Human and the absence
of disentangled datasets, we design a novel training scheme that creates
geometry-preserved and appearance-transferred training data to tune a generator
to achieve disentangled geometry and appearance control. Although our method is
trained with synthetic data, it can handle hand-drawn sketches as well.
Qualitative and quantitative evaluations demonstrate the superior performance
of our method to state-of-the-art methods.

这项工作提出了 Sketch2Human，这是第一个通过语义草图（用于几何控制）和参考图像（用于外观控制）引导的可控全身人体图像生成系统。

Sketch2Human：具有解耦几何和外观控制的深度人体生成

Sketch2Human: Deep Human Generation with Disentangled Geometry and  Appearance Control

Large-scale text-to-video (T2V) diffusion models have great progress in
recent years in terms of visual quality, motion and temporal consistency.
However, the generation process is still a black box, where all attributes
(e.g., appearance, motion) are learned and generated jointly without precise
control ability other than rough text descriptions. Inspired by image animation
which decouples the video as one specific appearance with the corresponding
motion, we propose AnimateZero to unveil the pre-trained text-to-video
diffusion model, i.e., AnimateDiff, and provide more precise appearance and
motion control abilities for it. For appearance control, we borrow intermediate
latents and their features from the text-to-image (T2I) generation for ensuring
the generated first frame is equal to the given generated image. For temporal
control, we replace the global temporal attention of the original T2V model
with our proposed positional-corrected window attention to ensure other frames
align with the first frame well. Empowered by the proposed methods, AnimateZero
can successfully control the generating progress without further training. As a
zero-shot image animator for given images, AnimateZero also enables multiple
new applications, including interactive video generation and real image
animation. The detailed experiments demonstrate the effectiveness of the
proposed method in both T2V and related applications.

通过解耦视频为特定外观与相应动作的图像动画，我们提出了 AnimateZero 来揭示预训练的文本到视频扩散模型 AnimateDiff，并为其提供更精确的外观和动作控制能力。Anima teZero 无需进一步训练即可成功控制生成进程，对于给定图像，还可以实现多种新应用，包括交互式视频生成和真实图像动画。详细实验表明，该方法在 T2V 和相关应用中具有有效性。