Creating a digital human avatar that is relightable, drivable, and
photorealistic is a challenging and important problem in Vision and Graphics.
Humans are highly articulated creating pose-dependent appearance effects like
self-shadows and wrinkles, and skin as well as clothing require complex and
space-varying BRDF models. While recent human relighting approaches can recover
plausible material-light decompositions from multi-view video, they do not
generalize to novel poses and still suffer from visual artifacts. To address
this, we propose Relightable Neural Actor, the first video-based method for
learning a photorealistic neural human model that can be relighted, allows
appearance editing, and can be controlled by arbitrary skeletal poses.
Importantly, for learning our human avatar, we solely require a multi-view
recording of the human under a known, but static lighting condition. To achieve
this, we represent the geometry of the actor with a drivable density field that
models pose-dependent clothing deformations and provides a mapping between 3D
and UV space, where normal, visibility, and materials are encoded. To evaluate
our approach in real-world scenarios, we collect a new dataset with four actors
recorded under different light conditions, indoors and outdoors, providing the
first benchmark of its kind for human relighting, and demonstrating
state-of-the-art relighting results for novel human poses.

通过基于多视角视频的学习方法，我们提出了可重光的神经网络人体模型，能够实现真实感的光照效果、外观编辑以及任意骨骼姿势控制，并在实际场景中进行了评估，展示了针对新颖人体动作的最新重光效果。

具备内在分解和姿势控制的可重新照明的神经行为模型

Relightable Neural Actor with Intrinsic Decomposition and Pose Control

Human relighting is a highly desirable yet challenging task. Existing works
either require expensive one-light-at-a-time (OLAT) captured data using light
stage or cannot freely change the viewpoints of the rendered body. In this
work, we propose a principled framework, Relighting4D, that enables
free-viewpoints relighting from only human videos under unknown illuminations.
Our key insight is that the space-time varying geometry and reflectance of the
human body can be decomposed as a set of neural fields of normal, occlusion,
diffuse, and specular maps. These neural fields are further integrated into
reflectance-aware physically based rendering, where each vertex in the neural
field absorbs and reflects the light from the environment. The whole framework
can be learned from videos in a self-supervised manner, with physically
informed priors designed for regularization. Extensive experiments on both real
and synthetic datasets demonstrate that our framework is capable of relighting
dynamic human actors with free-viewpoints.

本文提出了一个称之为 Relighting4D 的结构化框架，通过神经网络分解人体的空间时间几何和反射属性为一组法线、遮挡、漫反射和镜面映射的神经场，进而进行基于物理的面部渲染且可自我学习。通过在真实和合成数据集上的广泛实验验证，我们的方法能够以自我监督学习的方式将人体演员的表演与其背景分离，实现无限可调的动态换背景。