Most earlier investigations on talking face generation have focused on the
synchronization of lip motion and speech content. However, human head pose and
facial emotions are equally important characteristics of natural human faces.
While audio-driven talking face generation has seen notable advancements,
existing methods either overlook facial emotions or are limited to specific
individuals and cannot be applied to arbitrary subjects. In this paper, we
propose a one-shot Talking Head Generation framework (SPEAK) that distinguishes
itself from general Talking Face Generation by enabling emotional and postural
control. Specifically, we introduce the Inter-Reconstructed Feature
Disentanglement (IRFD) method to decouple human facial features into three
latent spaces. We then design a face editing module that modifies speech
content and facial latent codes into a single latent space. Subsequently, we
present a novel generator that employs modified latent codes derived from the
editing module to regulate emotional expression, head poses, and speech content
in synthesizing facial animations. Extensive trials demonstrate that our method
can generate realistic talking head with coordinated lip motions, authentic
facial emotions, and smooth head movements. The demo video is available at the
anonymous link: this https URL

提出了一种名为 SPEAK 的一次性 Talking Head Generation 框架，通过情感和姿势控制实现与一般 Talking Face Generation 的区别。该方法采用 Inter-Reconstructed Feature Disentanglement (IRFD) 方法将人脸特征解耦为三个潜在空间，并设计了一个面部编辑模块，将语音内容和面部潜在编码修改为单一的潜在空间。进一步，提出了一种新颖的生成器，利用编辑模块生成的修改后的潜在编码来调节情感表达、头部姿势和语音内容，以合成面部动画。大量实验表明，该方法可以生成具有协调的唇部运动、真实的面部情感和平滑的头部运动的逼真说话角色。

听、解缠与控制：可控语音驱动的说话人头像生成

Listen, Disentangle, and Control: Controllable Speech-Driven Talking  Head Generation

Animating virtual characters has always been a fundamental research problem
in virtual reality (VR). Facial animations play a crucial role as they
effectively convey emotions and attitudes of virtual humans. However, creating
such facial animations can be challenging, as current methods often involve
utilization of expensive motion capture devices or significant investments of
time and effort from human animators in tuning animation parameters. In this
paper, we propose a holistic solution to automatically animate virtual human
faces. In our solution, a deep learning model was first trained to retarget the
facial expression from input face images to virtual human faces by estimating
the blendshape coefficients. This method offers the flexibility of generating
animations with characters of different appearances and blendshape topologies.
Second, a practical toolkit was developed using Unity 3D, making it compatible
with the most popular VR applications. The toolkit accepts both image and video
as input to animate the target virtual human faces and enables users to
manipulate the animation results. Furthermore, inspired by the spirit of
Human-in-the-loop (HITL), we leveraged user feedback to further improve the
performance of the model and toolkit, thereby increasing the customization
properties to suit user preferences. The whole solution, for which we will make
the code public, has the potential to accelerate the generation of facial
animations for use in VR applications.

提出了一种全面的解决方案，通过深度学习模型将面部表情从输入面部图像转移到虚拟人脸，实现虚拟人物面部动画的自动化。使用 Unity 3D 开发了实用工具包，可接受图像和视频作为输入，以动画形式呈现目标虚拟人脸，并允许用户对动画结果进行操作。通过用户反馈不断改进模型和工具包的性能，从而增加了定制化属性以适应用户偏好。整个解决方案有助于加速用于虚拟现实应用的面部动画的生成。

自带角色：定制角色自动面部动画生成的整体解决方案

Bring Your Own Character: A Holistic Solution for Automatic Facial  Animation Generation of Customized Characters

Implementing fine-grained emotion control is crucial for emotion generation
tasks because it enhances the expressive capability of the generative model,
allowing it to accurately and comprehensively capture and express various
nuanced emotional states, thereby improving the emotional quality and
personalization of generated content. Generating fine-grained facial animations
that accurately portray emotional expressions using only a portrait and an
audio recording presents a challenge. In order to address this challenge, we
propose a visual attribute-guided audio decoupler. This enables the obtention
of content vectors solely related to the audio content, enhancing the stability
of subsequent lip movement coefficient predictions. To achieve more precise
emotional expression, we introduce a fine-grained emotion coefficient
prediction module. Additionally, we propose an emotion intensity control method
using a fine-grained emotion matrix. Through these, effective control over
emotional expression in the generated videos and finer classification of
emotion intensity are accomplished. Subsequently, a series of 3DMM coefficient
generation networks are designed to predict 3D coefficients, followed by the
utilization of a rendering network to generate the final video. Our
experimental results demonstrate that our proposed method, EmoSpeaker,
outperforms existing emotional talking face generation methods in terms of
expression variation and lip synchronization. Project page:
this https URL

通过引入细粒度情绪控制和音频分离模块来实现精确表达细致情绪状态的面部动画生成，有效控制情感表达并提高生成视频的表达变化和嘴唇同步性能。