In this paper, we aim at synthesizing a free-viewpoint video of an arbitrary
human performance using sparse multi-view cameras. Recently, several works have
addressed this problem by learning person-specific neural radiance fields
(NeRF) to capture the appearance of a particular human. In parallel, some work
proposed to use pixel-aligned features to generalize radiance fields to
arbitrary new scenes and objects. Adopting such generalization approaches to
humans, however, is highly challenging due to the heavy occlusions and dynamic
articulations of body parts. To tackle this, we propose Neural Human Performer,
a novel approach that learns generalizable neural radiance fields based on a
parametric human body model for robust performance capture. Specifically, we
first introduce a temporal transformer that aggregates tracked visual features
based on the skeletal body motion over time. Moreover, a multi-view transformer
is proposed to perform cross-attention between the temporally-fused features
and the pixel-aligned features at each time step to integrate observations on
the fly from multiple views. Experiments on the ZJU-MoCap and AIST datasets
show that our method significantly outperforms recent generalizable NeRF
methods on unseen identities and poses. The video results and code are
available at this https URL

本研究提出了一种基于人体参数模型学习可泛化神经辐射场的新方法，使用稀疏多视角相机合成任意人类表演的自由视点视频，并在实验中取得了明显的优于以往 NeRF 方法的表现。