Recent years have witnessed great progress in creating vivid audio-driven
portraits from monocular videos. However, how to seamlessly adapt the created
video avatars to other scenarios with different backgrounds and lighting
conditions remains unsolved. On the other hand, existing relighting studies
mostly rely on dynamically lighted or multi-view data, which are too expensive
for creating video portraits. To bridge this gap, we propose ReliTalk, a novel
framework for relightable audio-driven talking portrait generation from
monocular videos. Our key insight is to decompose the portrait's reflectance
from implicitly learned audio-driven facial normals and images. Specifically,
we involve 3D facial priors derived from audio features to predict delicate
normal maps through implicit functions. These initially predicted normals then
take a crucial part in reflectance decomposition by dynamically estimating the
lighting condition of the given video. Moreover, the stereoscopic face
representation is refined using the identity-consistent loss under simulated
multiple lighting conditions, addressing the ill-posed problem caused by
limited views available from a single monocular video. Extensive experiments
validate the superiority of our proposed framework on both real and synthetic
datasets. Our code is released in this https URL

从单眼视频中生成可闪烁的音频驱动的说话肖像的新框架，使用隐式学习的面部法线和图像，结合三维面部先验和动态估计视频的光线条件进行反射分解，并通过身份一致性损失改进立体面部表示。

ReliTalk：基于单一视频的可重光的说话肖像生成

ReliTalk: Relightable Talking Portrait Generation from a Single Video

State-of-the-art face super-resolution methods employ deep convolutional
neural networks to learn a mapping between low- and high- resolution facial
patterns by exploring local appearance knowledge. However, most of these
methods do not well exploit facial structures and identity information, and
struggle to deal with facial images that exhibit large pose variations. In this
paper, we propose a novel face super-resolution method that explicitly
incorporates 3D facial priors which grasp the sharp facial structures. Our work
is the first to explore 3D morphable knowledge based on the fusion of
parametric descriptions of face attributes (e.g., identity, facial expression,
texture, illumination, and face pose). Furthermore, the priors can easily be
incorporated into any network and are extremely efficient in improving the
performance and accelerating the convergence speed. Firstly, a 3D face
rendering branch is set up to obtain 3D priors of salient facial structures and
identity knowledge. Secondly, the Spatial Attention Module is used to better
exploit this hierarchical information (i.e., intensity similarity, 3D facial
structure, and identity content) for the super-resolution problem. Extensive
experiments demonstrate that the proposed 3D priors achieve superior face
super-resolution results over the state-of-the-arts.

该研究提出了一种将 3D 人脸先验知识与深度卷积神经网络相结合的人脸超分辨率方法。实验结果表明，所提出的方法在面部超分辨率方面取得了优异的效果。