We present the first approach to volumetric performance capture and
novel-view rendering at real-time speed from monocular video, eliminating the
need for expensive multi-view systems or cumbersome pre-acquisition of a
personalized template model. Our system reconstructs a fully textured 3D human
from each frame by leveraging Pixel-Aligned Implicit Function (PIFu). While
PIFu achieves high-resolution reconstruction in a memory-efficient manner, its
computationally expensive inference prevents us from deploying such a system
for real-time applications. To this end, we propose a novel hierarchical
surface localization algorithm and a direct rendering method without explicitly
extracting surface meshes. By culling unnecessary regions for evaluation in a
coarse-to-fine manner, we successfully accelerate the reconstruction by two
orders of magnitude from the baseline without compromising the quality.
Furthermore, we introduce an Online Hard Example Mining (OHEM) technique that
effectively suppresses failure modes due to the rare occurrence of challenging
examples. We adaptively update the sampling probability of the training data
based on the current reconstruction accuracy, which effectively alleviates
reconstruction artifacts. Our experiments and evaluations demonstrate the
robustness of our system to various challenging angles, illuminations, poses,
and clothing styles. We also show that our approach compares favorably with the
state-of-the-art monocular performance capture. Our proposed approach removes
the need for multi-view studio settings and enables a consumer-accessible
solution for volumetric capture.

本研究通过引入 PIFu 算法和使用表面本地化算法以及在线难例挖掘技术，从而在不需要昂贵的多视图系统或繁琐的预定义模型的情况下，从单眼视频中实时捕捉和渲染了一个完整的实纹三维人体。