Recent advances in generative diffusion models have enabled the previously
unfeasible capability of generating 3D assets from a single input image or a
text prompt. In this work, we aim to enhance the quality and functionality of
these models for the task of creating controllable, photorealistic human
avatars. We achieve this by integrating a 3D morphable model into the
state-of-the-art multiview-consistent diffusion approach. We demonstrate that
accurate conditioning of a generative pipeline on the articulated 3D model
enhances the baseline model performance on the task of novel view synthesis
from a single image. More importantly, this integration facilitates a seamless
and accurate incorporation of facial expression and body pose control into the
generation process. To the best of our knowledge, our proposed framework is the
first diffusion model to enable the creation of fully 3D-consistent,
animatable, and photorealistic human avatars from a single image of an unseen
subject; extensive quantitative and qualitative evaluations demonstrate the
advantages of our approach over existing state-of-the-art avatar creation
models on both novel view and novel expression synthesis tasks.

本研究工作旨在通过将 3D 可变模型整合到最新的多视角一致性扩散方法中，增强生成扩散模型在创建可控、照片般逼真的人类头像任务中的质量和功能。我们的实验证明了在基于关节的 3D 模型的准确约束下，生成流水线模型在单图像的新视角合成任务上的性能改进，更重要的是，这种整合实现了面部表情和身体姿势控制在生成过程中的无缝和准确融入。据我们所知，我们提出的框架是第一个允许从未见过的单一图像创建完全 3D 一致、可动画和照片般逼真的人类头像的扩散模型；广泛的定量和定性评估证明了我们的方法在新视角和新表情合成任务上相对于现有最先进的头像创建模型的优势。

可塑扩散：单图像化身创造的三维一致扩散

Morphable Diffusion: 3D-Consistent Diffusion for Single-image Avatar  Creation

In this paper, we propose a novel virtual try-on from unconstrained designs
(ucVTON) task to enable photorealistic synthesis of personalized composite
clothing on input human images. Unlike prior arts constrained by specific input
types, our method allows flexible specification of style (text or image) and
texture (full garment, cropped sections, or texture patches) conditions. To
address the entanglement challenge when using full garment images as
conditions, we develop a two-stage pipeline with explicit disentanglement of
style and texture. In the first stage, we generate a human parsing map
reflecting the desired style conditioned on the input. In the second stage, we
composite textures onto the parsing map areas based on the texture input. To
represent complex and non-stationary textures that have never been achieved in
previous fashion editing works, we first propose extracting hierarchical and
balanced CLIP features and applying position encoding in VTON. Experiments
demonstrate superior synthesis quality and personalization enabled by our
method. The flexible control over style and texture mixing brings virtual
try-on to a new level of user experience for online shopping and fashion
design.

本研究提出了一种创新的虚拟试穿技术，能够在输入人体图像上实现个性化服装的逼真合成。我们的方法具有灵活的样式和纹理条件，并通过明确分离样式和纹理的双阶段流程来解决全服装图像作为条件时的交织挑战。通过提取层次化和平衡的 CLIP 特征以及在 VTON 中应用位置编码，我们首次实现了复杂的非平稳纹理，实验结果表明了我们方法在合成质量和个性化方面的卓越表现。样式和纹理的灵活控制为在线购物和时尚设计提供了新的用户体验水平。

PICTURE: 非约束设计的光真实虚拟试穿

PICTURE: PhotorealistIC virtual Try-on from UnconstRained dEsigns

We present an approach to generate a 360-degree view of a person with a
consistent, high-resolution appearance from a single input image. NeRF and its
variants typically require videos or images from different viewpoints. Most
existing approaches taking monocular input either rely on ground-truth 3D scans
for supervision or lack 3D consistency. While recent 3D generative models show
promise of 3D consistent human digitization, these approaches do not generalize
well to diverse clothing appearances, and the results lack photorealism. Unlike
existing work, we utilize high-capacity 2D diffusion models pretrained for
general image synthesis tasks as an appearance prior of clothed humans. To
achieve better 3D consistency while retaining the input identity, we
progressively synthesize multiple views of the human in the input image by
inpainting missing regions with shape-guided diffusion conditioned on
silhouette and surface normal. We then fuse these synthesized multi-view images
via inverse rendering to obtain a fully textured high-resolution 3D mesh of the
given person. Experiments show that our approach outperforms prior methods and
achieves photorealistic 360-degree synthesis of a wide range of clothed humans
with complex textures from a single image.

通过将高容量的二维扩散模型与形状引导扩散相结合并利用逆渲染技术，从单一图像中逐步合成多个视角的完全纹理化高分辨率三维网格，实现了广泛的服装人物 360 度合成的照片级别结果。