When working with 3D facial data, improving fidelity and avoiding the uncanny
valley effect is critically dependent on accurate 3D facial performance
capture. Because such methods are expensive and due to the widespread
availability of 2D videos, recent methods have focused on how to perform
monocular 3D face tracking. However, these methods often fall short in
capturing precise facial movements due to limitations in their network
architecture, training, and evaluation processes. Addressing these challenges,
we propose a novel face tracker, FlowFace, that introduces an innovative 2D
alignment network for dense per-vertex alignment. Unlike prior work, FlowFace
is trained on high-quality 3D scan annotations rather than weak supervision or
synthetic data. Our 3D model fitting module jointly fits a 3D face model from
one or many observations, integrating existing neutral shape priors for
enhanced identity and expression disentanglement and per-vertex deformations
for detailed facial feature reconstruction. Additionally, we propose a novel
metric and benchmark for assessing tracking accuracy. Our method exhibits
superior performance on both custom and publicly available benchmarks. We
further validate the effectiveness of our tracker by generating high-quality 3D
data from 2D videos, which leads to performance gains on downstream tasks.

通过引入创新的 2D 对齐网络以进行稠密的顶点对齐，我们提出了一种新颖的人脸追踪器 FlowFace，用于改善三维面部数据的精确性和避免反感谷效应。与先前的工作不同，FlowFace 是基于高质量的三维扫描注释进行训练，而不是基于弱监督或合成数据。我们的方法在自定义和公开可用的基准测试中表现出优秀的性能，并通过从二维视频生成高质量的三维数据来验证追踪器的有效性，这导致在下游任务中获得性能提升。