We introduce Structured 3D Features, a model based on a novel implicit 3D
representation that pools pixel-aligned image features onto dense 3D points
sampled from a parametric, statistical human mesh surface. The 3D points have
associated semantics and can move freely in 3D space. This allows for optimal
coverage of the person of interest, beyond just the body shape, which in turn,
additionally helps modeling accessories, hair, and loose clothing. Owing to
this, we present a complete 3D transformer-based attention framework which,
given a single image of a person in an unconstrained pose, generates an
animatable 3D reconstruction with albedo and illumination decomposition, as a
result of a single end-to-end model, trained semi-supervised, and with no
additional postprocessing. We show that our S3F model surpasses the previous
state-of-the-art on various tasks, including monocular 3D reconstruction, as
well as albedo and shading estimation. Moreover, we show that the proposed
methodology allows novel view synthesis, relighting, and re-posing the
reconstruction, and can naturally be extended to handle multiple input images
(e.g. different views of a person, or the same view, in different poses, in
video). Finally, we demonstrate the editing capabilities of our model for 3D
virtual try-on applications.

本研究提出了一种基于隐式 3D 表示的 Structured 3D Features 模型，通过对参数统计人体网格表面进行密集的 3D 点采样并将像素对齐的图像特征汇集到其中，生成一个单端到端模型，可以生成带有颜色和光照分解的动画化 3D 重建图像，在 3D 虚拟试穿等应用领域有很大应用潜力。