The recent state of the art on monocular 3D face reconstruction from image
data has made some impressive advancements, thanks to the advent of Deep
Learning. However, it has mostly focused on input coming from a single RGB
image, overlooking the following important factors: a) Nowadays, the vast
majority of facial image data of interest do not originate from single images
but rather from videos, which contain rich dynamic information. b) Furthermore,
these videos typically capture individuals in some form of verbal communication
(public talks, teleconferences, audiovisual human-computer interactions,
interviews, monologues/dialogues in movies, etc). When existing 3D face
reconstruction methods are applied in such videos, the artifacts in the
reconstruction of the shape and motion of the mouth area are often severe,
since they do not match well with the speech audio.
To overcome the aforementioned limitations, we present the first method for
visual speech-aware perceptual reconstruction of 3D mouth expressions. We do
this by proposing a "lipread" loss, which guides the fitting process so that
the elicited perception from the 3D reconstructed talking head resembles that
of the original video footage. We demonstrate that, interestingly, the lipread
loss is better suited for 3D reconstruction of mouth movements compared to
traditional landmark losses, and even direct 3D supervision. Furthermore, the
devised method does not rely on any text transcriptions or corresponding audio,
rendering it ideal for training in unlabeled datasets. We verify the efficiency
of our method through exhaustive objective evaluations on three large-scale
datasets, as well as subjective evaluation with two web-based user studies.

本文介绍了一种基于视觉的、口型感知的三维口部表情重建方法，该方法不需要任何文本转录或对应音频，并使用 “lipread” 损失来指导拟合过程，以使三维重建的头部运动与原始视频片段的感知相似。此外，通过三个大规模数据集的详尽客观评估和两个基于网络的用户研究的主观评估，我们证明了该方法的有效性。