A long-standing goal of 3D human reconstruction is to create lifelike and
fully detailed 3D humans from single images. The main challenge lies in
inferring unknown human shapes, clothing, and texture information in areas not
visible in the images. To address this, we propose SiTH, a novel pipeline that
uniquely integrates an image-conditioned diffusion model into a 3D mesh
reconstruction workflow. At the core of our method lies the decomposition of
the ill-posed single-view reconstruction problem into hallucination and
reconstruction subproblems. For the former, we employ a powerful generative
diffusion model to hallucinate back appearances from the input images. For the
latter, we leverage skinned body meshes as guidance to recover full-body
texture meshes from the input and back-view images. Our designs enable training
of the pipeline with only about 500 3D human scans while maintaining its
generality and robustness. Extensive experiments and user studies on two 3D
reconstruction benchmarks demonstrated the efficacy of our method in generating
realistic, fully textured 3D humans from a diverse range of unseen images.

提出了一种新的流程 SiTH，该流程将基于图像的扩散模型与 3D 网格重建相结合，通过将单视图重建问题分解为幻觉和重建子问题来推断未知的人体形状、服装和纹理信息。该方法通过强大的生成扩散模型重新生成输入图像中的外貌，并借助有皮肤的身体网格作为指导从输入图像和背视图图像中恢复出完整的人体纹理网格。在仅使用约 500 个 3D 人体扫描进行训练的同时，保持了方法的普遍性和鲁棒性。在两个 3D 重建基准测试上进行了大量实验和用户研究，证明了该方法在从各种未见图像中生成逼真、带纹理的 3D 人体方面的有效性。