In this paper, we present a video-based learning framework for animating
personalized 3D talking faces from audio. We introduce two training-time data
normalizations that significantly improve data sample efficiency. First, we
isolate and represent faces in a normalized space that decouples 3D geometry,
head pose, and texture. This decomposes the prediction problem into regressions
over the 3D face shape and the corresponding 2D texture atlas. Second, we
leverage facial symmetry and approximate albedo constancy of skin to isolate
and remove spatio-temporal lighting variations. Together, these normalizations
allow simple networks to generate high fidelity lip-sync videos under novel
ambient illumination while training with just a single speaker-specific video.
Further, to stabilize temporal dynamics, we introduce an auto-regressive
approach that conditions the model on its previous visual state. Human ratings
and objective metrics demonstrate that our method outperforms contemporary
state-of-the-art audio-driven video reenactment benchmarks in terms of realism,
lip-sync and visual quality scores. We illustrate several applications enabled
by our framework.

本文提出了一种基于视频学习的框架，用于从音频中制作个性化的 3D 说话人脸，其中使用面部标准化和自回归方法来提高样本效率并生成高保真的口型同步视频。

LipSync3D：使用姿态和光照规范化从视频中高效学习个性化的三维说话脸

LipSync3D: Data-Efficient Learning of Personalized 3D Talking Faces from  Video using Pose and Lighting Normalization

We present ObamaNet, the first architecture that generates both audio and
synchronized photo-realistic lip-sync videos from any new text. Contrary to
other published lip-sync approaches, ours is only composed of fully trainable
neural modules and does not rely on any traditional computer graphics methods.
More precisely, we use three main modules: a text-to-speech network based on
Char2Wav, a time-delayed LSTM to generate mouth-keypoints synced to the audio,
and a network based on Pix2Pix to generate the video frames conditioned on the
keypoints.

本文介绍了 ObamaNet 这一模型，该模型利用全可训练的神经模块，生成新文本的音频文件和同步的逼真唇形视频，其中所涉及的三个主要模块分别是基于 Char2Wav 的文本转语音网络、生成与音频同步的嘴部关键点的时延 LSTM 网络、以及基于 Pix2Pix 的条件渲染视频帧的网络。