We present the first approach to render highly realistic free-viewpoint
videos of a human actor in general apparel, from sparse multi-view recording to
display, in real-time at an unprecedented 4K resolution. At inference, our
method only requires four camera views of the moving actor and the respective
3D skeletal pose. It handles actors in wide clothing, and reproduces even
fine-scale dynamic detail, e.g. clothing wrinkles, face expressions, and hand
gestures. At training time, our learning-based approach expects dense
multi-view video and a rigged static surface scan of the actor. Our method
comprises three main stages. Stage 1 is a skeleton-driven neural approach for
high-quality capture of the detailed dynamic mesh geometry. Stage 2 is a novel
solution to create a view-dependent texture using four test-time camera views
as input. Finally, stage 3 comprises a new image-based refinement network
rendering the final 4K image given the output from the previous stages. Our
approach establishes a new benchmark for real-time rendering resolution and
quality using sparse input camera views, unlocking possibilities for immersive
telepresence.

我们提出了第一种方法，可以实时以前所未有的 4K 分辨率渲染穿着常规服装的人体演员的极其逼真的自由视点视频。

稀疏 RGB 相机实时自由视点渲染的全息传输人物

Holoported Characters: Real-time Free-viewpoint Rendering of Humans from  Sparse RGB Cameras

We present the first real-time method to capture the full global 3D skeletal
pose of a human in a stable, temporally consistent manner using a single RGB
camera. Our method combines a new convolutional neural network (CNN) based pose
regressor with kinematic skeleton fitting. Our novel fully-convolutional pose
formulation regresses 2D and 3D joint positions jointly in real time and does
not require tightly cropped input frames. A real-time kinematic skeleton
fitting method uses the CNN output to yield temporally stable 3D global pose
reconstructions on the basis of a coherent kinematic skeleton. This makes our
approach the first monocular RGB method usable in real-time applications such
as 3D character control---thus far, the only monocular methods for such
applications employed specialized RGB-D cameras. Our method's accuracy is
quantitatively on par with the best offline 3D monocular RGB pose estimation
methods. Our results are qualitatively comparable to, and sometimes better
than, results from monocular RGB-D approaches, such as the Kinect. However, we
show that our approach is more broadly applicable than RGB-D solutions, i.e. it
works for outdoor scenes, community videos, and low quality commodity RGB
cameras.

采用单一 RGB 相机，我们提出了第一个实时方法来稳定地连续捕捉人类的全局三维骨架姿态，并结合卷积神经网络和运动学骨骼拟合，成功实现了单目 RGB 方法，避免了使用专门的 RGB-D 相机，同时具有优越的性能和实用价值。