Creating high-quality controllable 3D human models from multi-view RGB videos
poses a significant challenge. Neural radiance fields (NeRFs) have demonstrated
remarkable quality in reconstructing and free-viewpoint rendering of static as
well as dynamic scenes. The extension to a controllable synthesis of dynamic
human performances poses an exciting research question. In this paper, we
introduce a novel NeRF-based framework for pose-dependent rendering of human
performances. In our approach, the radiance field is warped around an SMPL body
mesh, thereby creating a new surface-aligned representation. Our representation
can be animated through skeletal joint parameters that are provided to the NeRF
in addition to the viewpoint for pose dependent appearances. To achieve this,
our representation includes the corresponding 2D UV coordinates on the mesh
texture map and the distance between the query point and the mesh. To enable
efficient learning despite mapping ambiguities and random visual variations, we
introduce a novel remapping process that refines the mapped coordinates.
Experiments demonstrate that our approach results in high-quality renderings
for novel-view and novel-pose synthesis.

基于 NeRF 的人体表演姿态依赖渲染的新方法，通过将辐射场围绕 SMPL 人体模型进行扭曲，实现了新的表面对齐表示，可以通过骨骼关节参数和视点来实现动画化，并通过引入新的重新映射过程来实现高质量渲染。

从纹理空间中为 NeRFs 进行动画：一种基于姿态的人类表演渲染框架

Animating NeRFs from Texture Space: A Framework for Pose-Dependent  Rendering of Human Performances

We have recently seen tremendous progress in the neural advances for
photo-real human modeling and rendering. However, it's still challenging to
integrate them into an existing mesh-based pipeline for downstream
applications. In this paper, we present a comprehensive neural approach for
high-quality reconstruction, compression, and rendering of human performances
from dense multi-view videos. Our core intuition is to bridge the traditional
animated mesh workflow with a new class of highly efficient neural techniques.
We first introduce a neural surface reconstructor for high-quality surface
generation in minutes. It marries the implicit volumetric rendering of the
truncated signed distance field (TSDF) with multi-resolution hash encoding. We
further propose a hybrid neural tracker to generate animated meshes, which
combines explicit non-rigid tracking with implicit dynamic deformation in a
self-supervised framework. The former provides the coarse warping back into the
canonical space, while the latter implicit one further predicts the
displacements using the 4D hash encoding as in our reconstructor. Then, we
discuss the rendering schemes using the obtained animated meshes, ranging from
dynamic texturing to lumigraph rendering under various bandwidth settings. To
strike an intricate balance between quality and bandwidth, we propose a
hierarchical solution by first rendering 6 virtual views covering the performer
and then conducting occlusion-aware neural texture blending. We demonstrate the
efficacy of our approach in a variety of mesh-based applications and
photo-realistic free-view experiences on various platforms, i.e., inserting
virtual human performances into real environments through mobile AR or
immersively watching talent shows with VR headsets.

本文提出了一种综合的神经方法，用于从密集的多视角视频中重建、压缩和渲染人类表演。该方法桥接了传统的动画网格工作流和一类高效的神经技术，其中包括神经表面重构器和混合神经跟踪器，并实现了各种带宽设置下的从动态纹理到光图渲染的渲染方案，展示了其在各种网格应用和各种平台的逼真自由视点体验中的有效性。