Capturing and faithfully rendering photo-realistic humans from novel views is
a fundamental problem for AR/VR applications. While prior work has shown
impressive performance capture results in laboratory settings, it is
non-trivial to achieve casual free-viewpoint human capture and rendering for
unseen identities with high fidelity, especially for facial expressions, hands,
and clothes. To tackle these challenges we introduce a novel view synthesis
framework that generates realistic renders from unseen views of any human
captured from a single-view and sparse RGB-D sensor, similar to a low-cost
depth camera, and without actor-specific models. We propose an architecture to
create dense feature maps in novel views obtained by sphere-based neural
rendering, and create complete renders using a global context inpainting model.
Additionally, an enhancer network leverages the overall fidelity, even in
occluded areas from the original view, producing crisp renders with fine
details. We show that our method generates high-quality novel views of
synthetic and real human actors given a single-stream, sparse RGB-D input. It
generalizes to unseen identities, and new poses and faithfully reconstructs
facial expressions. Our approach outperforms prior view synthesis methods and
is robust to different levels of depth sparsity.

提出了一种利用低成本深度相机从单视角和稀疏 RGB-D 传感器中捕获任意人物，并从未见过的视角生成逼真渲染的视图合成框架。该方法可重建面部表情，具有良好的鲁棒性和高质量的渲染效果，优于之前的视图合成方法。