4D reconstruction of human-object interaction is critical for immersive VR/AR
experience and human activity understanding. Recent advances still fail to
recover fine geometry and texture results from sparse RGB inputs, especially
under challenging human-object interactions scenarios. In this paper, we
propose a neural human performance capture and rendering system to generate
both high-quality geometry and photo-realistic texture of both human and
objects under challenging interaction scenarios in arbitrary novel views, from
only sparse RGB streams. To deal with complex occlusions raised by human-object
interactions, we adopt a layer-wise scene decoupling strategy and perform
volumetric reconstruction and neural rendering of the human and object.
Specifically, for geometry reconstruction, we propose an interaction-aware
human-object capture scheme that jointly considers the human reconstruction and
object reconstruction with their correlations. Occlusion-aware human
reconstruction and robust human-aware object tracking are proposed for
consistent 4D human-object dynamic reconstruction. For neural texture
rendering, we propose a layer-wise human-object rendering scheme, which
combines direction-aware neural blending weight learning and spatial-temporal
texture completion to provide high-resolution and photo-realistic texture
results in the occluded scenarios. Extensive experiments demonstrate the
effectiveness of our approach to achieve high-quality geometry and texture
reconstruction in free viewpoints for challenging human-object interactions.

本论文提出了一种神经人体表现捕捉和渲染系统，以实现高质量的几何和逼真的纹理重建，尤其是在挑战性的人体物体交互场景下。通过执行分层场景解耦策略，同时进行体积重建和神经渲染人和物体，提出了交互感知的人 - 物捕捉方案，同时提出了以人为重点的物体跟踪，在自由视点下实现了高质量的几何和纹理重建。