Creating realistic 3D facial animation is crucial for various applications in
the movie production and gaming industry, especially with the burgeoning demand
in the metaverse. However, prevalent methods such as blendshape-based
approaches and facial rigging techniques are time-consuming, labor-intensive,
and lack standardized configurations, making facial animation production
challenging and costly. In this paper, we propose a novel self-supervised
framework, Versatile Face Animator, which combines facial motion capture with
motion retargeting in an end-to-end manner, eliminating the need for
blendshapes or rigs. Our method has the following two main characteristics: 1)
we propose an RGBD animation module to learn facial motion from raw RGBD videos
by hierarchical motion dictionaries and animate RGBD images rendered from 3D
facial mesh coarse-to-fine, enabling facial animation on arbitrary 3D
characters regardless of their topology, textures, blendshapes, and rigs; and
2) we introduce a mesh retarget module to utilize RGBD animation to create 3D
facial animation by manipulating facial mesh with controller transformations,
which are estimated from dense optical flow fields and blended together with
geodesic-distance-based weights. Comprehensive experiments demonstrate the
effectiveness of our proposed framework in generating impressive 3D facial
animation results, highlighting its potential as a promising solution for the
cost-effective and efficient production of facial animation in the metaverse.

提出了一种新颖的自监督框架，名为 Versatile Face Animator，它将面部动作捕捉与动作重新定位结合起来，从而实现了在任意 3D 角色上进行面部动画的理想效果，并具备潜在的成本效益和高效率的潜力。