Volumetric (4D) performance capture is fundamental for AR/VR content
generation. Whereas previous work in 4D performance capture has shown
impressive results in studio settings, the technology is still far from being
accessible to a typical consumer who, at best, might own a single RGBD sensor.
Thus, in this work, we propose a method to synthesize free viewpoint renderings
using a single RGBD camera. The key insight is to leverage previously seen
"calibration" images of a given user to extrapolate what should be rendered in
a novel viewpoint from the data available in the sensor. Given these past
observations from multiple viewpoints, and the current RGBD image from a fixed
view, we propose an end-to-end framework that fuses both these data sources to
generate novel renderings of the performer. We demonstrate that the method can
produce high fidelity images, and handle extreme changes in subject pose and
camera viewpoints. We also show that the system generalizes to performers not
seen in the training data. We run exhaustive experiments demonstrating the
effectiveness of the proposed semi-parametric model (i.e. calibration images
available to the neural network) compared to other state of the art machine
learned solutions. Further, we compare the method with more traditional
pipelines that employ multi-view capture. We show that our framework is able to
achieve compelling results, with substantially less infrastructure than
previously required.

提出使用一台 RGBD 相机综合已获取的图像数据生成自由视角视频的方法。该方法通过半参数模型结合深度神经网络的技术实现 4D 表演捕捉，从而为 AR / VR 内容生成提供高逼真度的渲染效果。