Dense 3D facial motion capture from only monocular in-the-wild pairs of RGB
images is a highly challenging problem with numerous applications, ranging from
facial expression recognition to facial reenactment. In this work, we propose
DeepFaceFlow, a robust, fast, and highly-accurate framework for the dense
estimation of 3D non-rigid facial flow between pairs of monocular images. Our
DeepFaceFlow framework was trained and tested on two very large-scale facial
video datasets, one of them of our own collection and annotation, with the aid
of occlusion-aware and 3D-based loss function. We conduct comprehensive
experiments probing different aspects of our approach and demonstrating its
improved performance against state-of-the-art flow and 3D reconstruction
methods. Furthermore, we incorporate our framework in a full-head
state-of-the-art facial video synthesis method and demonstrate the ability of
our method in better representing and capturing the facial dynamics, resulting
in a highly-realistic facial video synthesis. Given registered pairs of images,
our framework generates 3D flow maps at ~60 fps.

DeepFaceFlow 是一种基于 3D 的深度学习框架，用于快速准确地从单目图像中捕捉面部运动，并且可以应用于面部表情识别等领域，同时结合了 occlusion-aware 和 3D-based 损失函数进行训练和测试，方法优于现有的流和 3D 方法，还能作为完整的头部面部视频合成中的一部分。

深度脸部流：野外稠密三维面部运动估计

DeepFaceFlow: In-the-wild Dense 3D Facial Motion Estimation

We present Neural Voice Puppetry, a novel approach for audio-driven facial
video synthesis. Given an audio sequence of a source person or digital
assistant, we generate a photo-realistic output video of a target person that
is in sync with the audio of the source input. This audio-driven facial
reenactment is driven by a deep neural network that employs a latent 3D face
model space. Through the underlying 3D representation, the model inherently
learns temporal stability while we leverage neural rendering to generate
photo-realistic output frames. Our approach generalizes across different
people, allowing us to synthesize videos of a target actor with the voice of
any unknown source actor or even synthetic voices that can be generated
utilizing standard text-to-speech approaches. Neural Voice Puppetry has a
variety of use-cases, including audio-driven video avatars, video dubbing, and
text-driven video synthesis of a talking head. We demonstrate the capabilities
of our method in a series of audio- and text-based puppetry examples, including
comparisons to state-of-the-art techniques and a user study.

本文介绍了神经语音操纵 (Neural Voice Puppetry) 的新方法，即通过深度神经网络驱动的三维面部模型，实现了基于音频和文本的视频合成，包括生成音频驱动的虚拟形象和文本驱动的谈话头像等多种用途。