We present dynamic neural radiance fields for modeling the appearance and
dynamics of a human face. Digitally modeling and reconstructing a talking human
is a key building-block for a variety of applications. Especially, for
telepresence applications in AR or VR, a faithful reproduction of the
appearance including novel viewpoints or head-poses is required. In contrast to
state-of-the-art approaches that model the geometry and material properties
explicitly, or are purely image-based, we introduce an implicit representation
of the head based on scene representation networks. To handle the dynamics of
the face, we combine our scene representation network with a low-dimensional
morphable model which provides explicit control over pose and expressions. We
use volumetric rendering to generate images from this hybrid representation and
demonstrate that such a dynamic neural scene representation can be learned from
monocular input data only, without the need of a specialized capture setup. In
our experiments, we show that this learned volumetric representation allows for
photo-realistic image generation that surpasses the quality of state-of-the-art
video-based reenactment methods.

使用动态神经辐射场模型重新构建并生成能够通过单目输入数据学习的场景表示，结合场景表示网络和低维可变形模型进行虚拟场景生成。

单目 4D 面部头像动态神经辐射场重建

Dynamic Neural Radiance Fields for Monocular 4D Facial Avatar  Reconstruction

We present an end-to-end neural network-based model for inferring an
approximate 3D mesh representation of a human face from single camera input for
AR applications. The relatively dense mesh model of 468 vertices is well-suited
for face-based AR effects. The proposed model demonstrates super-realtime
inference speed on mobile GPUs (100-1000+ FPS, depending on the device and
model variant) and a high prediction quality that is comparable to the variance
in manual annotations of the same image.

本文提出了一种基于神经网络的端到端模型，从单个摄像头输入中推断出人脸的近似 3D 网格表示，适用于 AR 应用程序。该模型具有相对密集的 468 个顶点网格模型，非常适合基于人脸的 AR 效果。该模型在移动 GPU 上具有超级实时推断速度（100-1000+FPS，取决于设备和模型变体），并且预测质量与同一图像的手动注释的变化相当。

使用移动 GPU 从单目视频实时获取面部表面几何信息

Real-time Facial Surface Geometry from Monocular Video on Mobile GPUs

With systems for acquiring 3D surface data being evermore commonplace, it has
become important to reliably extract specific shapes from the acquired data. In
the presence of noise and occlusions, this can be done through the use of
statistical shape models, which are learned from databases of clean examples of
the shape in question. In this paper, we review, analyze and compare different
statistical models: from those that analyze the variation in geometry globally
to those that analyze the variation in geometry locally. We first review how
different types of models have been used in the literature, then proceed to
define the models and analyze them theoretically, in terms of both their
statistical and computational aspects. We then perform extensive experimental
comparison on the task of model fitting, and give intuition about which type of
model is better for a few applications. Due to the wide availability of
databases of high-quality data, we use the human face as the specific shape we
wish to extract from corrupted data.

通过学习干净的人脸数据库和不同的统计形状模型，本文比较了全局和局部的几何形状变化分析，实验结果可以帮助我们在从降噪和遮挡的数据中可靠地提取特定形状时选择更好的模型。