We present dynamic neural radiance fields for modeling the appearance and
dynamics of a human face. Digitally modeling and reconstructing a talking human
is a key building-block for a variety of applications. Especially, for
telepresence applications in AR or VR, a faithful reproduction of the
appearance including novel viewpoints or head-poses is required. In contrast to
state-of-the-art approaches that model the geometry and material properties
explicitly, or are purely image-based, we introduce an implicit representation
of the head based on scene representation networks. To handle the dynamics of
the face, we combine our scene representation network with a low-dimensional
morphable model which provides explicit control over pose and expressions. We
use volumetric rendering to generate images from this hybrid representation and
demonstrate that such a dynamic neural scene representation can be learned from
monocular input data only, without the need of a specialized capture setup. In
our experiments, we show that this learned volumetric representation allows for
photo-realistic image generation that surpasses the quality of state-of-the-art
video-based reenactment methods.

使用动态神经辐射场模型重新构建并生成能够通过单目输入数据学习的场景表示，结合场景表示网络和低维可变形模型进行虚拟场景生成。

单目 4D 面部头像动态神经辐射场重建

Dynamic Neural Radiance Fields for Monocular 4D Facial Avatar  Reconstruction

For bidirectional joint image-text modeling, we develop variational
hetero-encoder (VHE) randomized generative adversarial network (GAN), a
versatile deep generative model that integrates a probabilistic text decoder,
probabilistic image encoder, and GAN into a coherent end-to-end multi-modality
learning framework. VHE randomized GAN (VHE-GAN) encodes an image to decode its
associated text, and feeds the variational posterior as the source of
randomness into the GAN image generator. We plug three off-the-shelf modules,
including a deep topic model, a ladder-structured image encoder, and
StackGAN++, into VHE-GAN, which already achieves competitive performance. This
further motivates the development of VHE-raster-scan-GAN that generates
photo-realistic images in not only a multi-scale low-to-high-resolution manner,
but also a hierarchical-semantic coarse-to-fine fashion. By capturing and
relating hierarchical semantic and visual concepts with end-to-end training,
VHE-raster-scan-GAN achieves state-of-the-art performance in a wide variety of
image-text multi-modality learning and generation tasks.

本文提出了可变异率异质编码器随机生成对抗网络 (VHE-GAN)，并将其应用于双向联合图像文本建模，以生成更准确的图像及关联文本，同时通过对抗训练和可变后验进行端到端训练来捕捉和关联层次语义和视觉概念，从而在各类图片和文本生成任务中取得了最先进的性能。