We propose NormalGAN, a fast adversarial learning-based method to reconstruct
the complete and detailed 3D human from a single RGB-D image. Given a single
front-view RGB-D image, NormalGAN performs two steps: front-view RGB-D
rectification and back-view RGBD inference. The final model was then generated
by simply combining the front-view and back-view RGB-D information. However,
inferring backview RGB-D image with high-quality geometric details and
plausible texture is not trivial. Our key observation is: Normal maps generally
encode much more information of 3D surface details than RGB and depth images.
Therefore, learning geometric details from normal maps is superior than other
representations. In NormalGAN, an adversarial learning framework conditioned by
normal maps is introduced, which is used to not only improve the front-view
depth denoising performance, but also infer the back-view depth image with
surprisingly geometric details. Moreover, for texture recovery, we remove
shading information from the front-view RGB image based on the refined normal
map, which further improves the quality of the back-view color inference.
Results and experiments on both testing data set and real captured data
demonstrate the superior performance of our approach. Given a consumer RGB-D
sensor, NormalGAN can generate the complete and detailed 3D human
reconstruction results in 20 fps, which further enables convenient interactive
experiences in telepresence, AR/VR and gaming scenarios.

这篇论文提出了一种名为 NormalGAN 的方法，使用对立学习的方式，从单张 RGB-D 图像中重建完整且详细的 3D 人体模型，该方法利用 Normal maps 编码的 3D 表面细节信息能够更好地学习几何细节，其性能优于其他表示方式，并且能够以 20fps 的速度在消费类 RGB-D 传感器上生成完整且详细的 3D 人体重建结果。