Recent progress in human shape learning, shows that neural implicit models
are effective in generating 3D human surfaces from limited number of views, and
even from a single RGB image. However, existing monocular approaches still
struggle to recover fine geometric details such as face, hands or cloth
wrinkles. They are also easily prone to depth ambiguities that result in
distorted geometries along the camera optical axis. In this paper, we explore
the benefits of incorporating depth observations in the reconstruction process
by introducing ANIM, a novel method that reconstructs arbitrary 3D human shapes
from single-view RGB-D images with an unprecedented level of accuracy. Our
model learns geometric details from both multi-resolution pixel-aligned and
voxel-aligned features to leverage depth information and enable spatial
relationships, mitigating depth ambiguities. We further enhance the quality of
the reconstructed shape by introducing a depth-supervision strategy, which
improves the accuracy of the signed distance field estimation of points that
lie on the reconstructed surface. Experiments demonstrate that ANIM outperforms
state-of-the-art works that use RGB, surface normals, point cloud or RGB-D data
as input. In addition, we introduce ANIM-Real, a new multi-modal dataset
comprising high-quality scans paired with consumer-grade RGB-D camera, and our
protocol to fine-tune ANIM, enabling high-quality reconstruction from
real-world human capture.

通过引入深度观察来重建任意 3D 人体形状的 ANIM 方法，在生成具有精确度的 3D 人体表面方面，优于使用 RGB、表面法线、点云或 RGB-D 数据的最新方法。

ANIM：从单个 RGB-D 图像准确重建人类模型的神经隐式模型

ANIM: Accurate Neural Implicit Model for Human Reconstruction from a  single RGB-D image

Large-scale pre-trained vision-language models allow for the zero-shot
text-based generation of 3D avatars. The previous state-of-the-art method
utilized CLIP to supervise neural implicit models that reconstructed a human
body mesh. However, this approach has two limitations. Firstly, the lack of
avatar-specific models can cause facial distortion and unrealistic clothing in
the generated avatars. Secondly, CLIP only provides optimization direction for
the overall appearance, resulting in less impressive results. To address these
limitations, we propose AvatarFusion, the first framework to use a latent
diffusion model to provide pixel-level guidance for generating human-realistic
avatars while simultaneously segmenting clothing from the avatar's body.
AvatarFusion includes the first clothing-decoupled neural implicit avatar model
that employs a novel Dual Volume Rendering strategy to render the decoupled
skin and clothing sub-models in one space. We also introduce a novel
optimization method, called Pixel-Semantics Difference-Sampling (PS-DS), which
semantically separates the generation of body and clothes, and generates a
variety of clothing styles. Moreover, we establish the first benchmark for
zero-shot text-to-avatar generation. Our experimental results demonstrate that
our framework outperforms previous approaches, with significant improvements
observed in all metrics. Additionally, since our model is clothing-decoupled,
we can exchange the clothes of avatars. Code will be available on Github.

AvatarFusion 利用一个潜在的扩散模型，同时从头到脚地生成人类逼真的头像，并将衣服与人物身体分割开来。AvatarFusion 包括第一个脱离衣物的神经隐式头像模型，该模型采用了新的双体积渲染策略，将脱离衣物和衣物子模型在一个空间中渲染。我们还引入了一种新的优化方法，称为 Pixel-Semantics Difference-Sampling（PS-DS），该方法在语义上将身体和衣物的生成分开，并生成各种服装风格。此外，由于我们的模型脱离了衣物，因此我们可以更换头像的衣服。