We present the first approach to render highly realistic free-viewpoint
videos of a human actor in general apparel, from sparse multi-view recording to
display, in real-time at an unprecedented 4K resolution. At inference, our
method only requires four camera views of the moving actor and the respective
3D skeletal pose. It handles actors in wide clothing, and reproduces even
fine-scale dynamic detail, e.g. clothing wrinkles, face expressions, and hand
gestures. At training time, our learning-based approach expects dense
multi-view video and a rigged static surface scan of the actor. Our method
comprises three main stages. Stage 1 is a skeleton-driven neural approach for
high-quality capture of the detailed dynamic mesh geometry. Stage 2 is a novel
solution to create a view-dependent texture using four test-time camera views
as input. Finally, stage 3 comprises a new image-based refinement network
rendering the final 4K image given the output from the previous stages. Our
approach establishes a new benchmark for real-time rendering resolution and
quality using sparse input camera views, unlocking possibilities for immersive
telepresence.

我们提出了第一种方法，可以实时以前所未有的 4K 分辨率渲染穿着常规服装的人体演员的极其逼真的自由视点视频。