Representing human performance at high-fidelity is an essential building
block in diverse applications, such as film production, computer games or
videoconferencing. To close the gap to production-level quality, we introduce
HumanRF, a 4D dynamic neural scene representation that captures full-body
appearance in motion from multi-view video input, and enables playback from
novel, unseen viewpoints. Our novel representation acts as a dynamic video
encoding that captures fine details at high compression rates by factorizing
space-time into a temporal matrix-vector decomposition. This allows us to
obtain temporally coherent reconstructions of human actors for long sequences,
while representing high-resolution details even in the context of challenging
motion. While most research focuses on synthesizing at resolutions of 4MP or
lower, we address the challenge of operating at 12MP. To this end, we introduce
ActorsHQ, a novel multi-view dataset that provides 12MP footage from 160
cameras for 16 sequences with high-fidelity, per-frame mesh reconstructions. We
demonstrate challenges that emerge from using such high-resolution data and
show that our newly introduced HumanRF effectively leverages this data, making
a significant step towards production-level quality novel view synthesis.

我们引入了 HumanRF，这是一种 4D 动态神经场景表示，可以捕获多视角视频输入中运动的全身外观，并允许从新的未见视角进行播放。通过分解时空为时间矩阵 - 向量分解，我们可以以高压缩比捕获高分辨率细节，并获得长序列的时间上连贯的演员重建。新的多视角数据集 ActorsHQ 提供高保真度的 16 个序列的 160 个相机的 12MP 镜头素材。我们的 HumanRF 有效利用了这些数据，在新颖视角合成方面取得了重要进展，实现接近于制作水平的品质。