A long-standing goal of 3D human reconstruction is to create lifelike and
fully detailed 3D humans from single images. The main challenge lies in
inferring unknown human shapes, clothing, and texture information in areas not
visible in the images. To address this, we propose SiTH, a novel pipeline that
uniquely integrates an image-conditioned diffusion model into a 3D mesh
reconstruction workflow. At the core of our method lies the decomposition of
the ill-posed single-view reconstruction problem into hallucination and
reconstruction subproblems. For the former, we employ a powerful generative
diffusion model to hallucinate back appearances from the input images. For the
latter, we leverage skinned body meshes as guidance to recover full-body
texture meshes from the input and back-view images. Our designs enable training
of the pipeline with only about 500 3D human scans while maintaining its
generality and robustness. Extensive experiments and user studies on two 3D
reconstruction benchmarks demonstrated the efficacy of our method in generating
realistic, fully textured 3D humans from a diverse range of unseen images.

提出了一种新的流程 SiTH，该流程将基于图像的扩散模型与 3D 网格重建相结合，通过将单视图重建问题分解为幻觉和重建子问题来推断未知的人体形状、服装和纹理信息。该方法通过强大的生成扩散模型重新生成输入图像中的外貌，并借助有皮肤的身体网格作为指导从输入图像和背视图图像中恢复出完整的人体纹理网格。在仅使用约 500 个 3D 人体扫描进行训练的同时，保持了方法的普遍性和鲁棒性。在两个 3D 重建基准测试上进行了大量实验和用户研究，证明了该方法在从各种未见图像中生成逼真、带纹理的 3D 人体方面的有效性。

SiTH：基于图像条件扩散的单视图纹理人体重建

SiTH: Single-view Textured Human Reconstruction with Image-Conditioned  Diffusion

We report Zero123++, an image-conditioned diffusion model for generating
3D-consistent multi-view images from a single input view. To take full
advantage of pretrained 2D generative priors, we develop various conditioning
and training schemes to minimize the effort of finetuning from off-the-shelf
image diffusion models such as Stable Diffusion. Zero123++ excels in producing
high-quality, consistent multi-view images from a single image, overcoming
common issues like texture degradation and geometric misalignment. Furthermore,
we showcase the feasibility of training a ControlNet on Zero123++ for enhanced
control over the generation process. The code is available at
this https URL

使用 Zero123++，一种图像条件扩散模型，从单个输入视图生成 3D 一致的多视图图像。通过开发各种调整和训练方案，最小化从现成的图像扩散模型（如稳定扩散）微调的工作量，以充分利用预先训练的 2D 生成先验。Zero123++ 在从单张图像生成高质量、一致的多视图图像方面表现卓越，克服了纹理退化和几何错位等常见问题。此外，我们展示了使用 Zero123++ 训练 ControlNet 来增强生成过程控制的可行性。代码可在此 https URL 处获得。

Zero123++: 一种将单张图像转化为一致多视角扩散基模型

Zero123++: a Single Image to Consistent Multi-view Diffusion Base Model

We propose a method for synthesizing edited photo-realistic digital avatars
with text instructions. Given a short monocular RGB video and text
instructions, our method uses an image-conditioned diffusion model to edit one
head image and uses the video stylization method to accomplish the editing of
other head images. Through iterative training and update (three times or more),
our method synthesizes edited photo-realistic animatable 3D neural head avatars
with a deformable neural radiance field head synthesis method. In quantitative
and qualitative studies on various subjects, our method outperforms
state-of-the-art methods.

本文提出一种使用图像条件扩散模型和神经辐射场方法，通过短单眼 RGB 视频和文本说明合成编辑后的、照片逼真的可动 3D 神经头像的方法。在各种主题的定量和定性研究中，我们的方法胜过了现有技术。