Audio-driven talking face generation has garnered significant interest within
the domain of digital human research. Existing methods are encumbered by
intricate model architectures that are intricately dependent on each other,
complicating the process of re-editing image or video inputs. In this work, we
present ControlTalk, a talking face generation method to control face
expression deformation based on driven audio, which can construct the head pose
and facial expression including lip motion for both single image or sequential
video inputs in a unified manner. By utilizing a pre-trained video synthesis
renderer and proposing the lightweight adaptation, ControlTalk achieves precise
and naturalistic lip synchronization while enabling quantitative control over
mouth opening shape. Our experiments show that our method is superior to
state-of-the-art performance on widely used benchmarks, including HDTF and
MEAD. The parameterized adaptation demonstrates remarkable generalization
capabilities, effectively handling expression deformation across same-ID and
cross-ID scenarios, and extending its utility to out-of-domain portraits,
regardless of languages.

基于音频驱动的控制型对话生成系统，可根据音频控制面部表情变形，包括单幅图像或顺序视频输入，能够实现准确而自然的口型同步，并能够定量控制嘴巴张开的形状。在广泛使用的基准测试中，我们的实验表明我们的方法在性能上优于最先进的技术，可实现表情变形的跨身份和跨语种的处理，并扩展其在异域肖像中的使用。

通过隐式面部关键点编辑实现可控的说话人脸生成

Controllable Talking Face Generation by Implicit Facial Keypoints  Editing

Since the beginning of the COVID-19 pandemic, remote conferencing and
school-teaching have become important tools. The previous applications aim to
save the commuting cost with real-time interactions. However, our application
is going to lower the production and reproduction costs when preparing the
communication materials. This paper proposes a system called Pre-Avatar,
generating a presentation video with a talking face of a target speaker with 1
front-face photo and a 3-minute voice recording. Technically, the system
consists of three main modules, user experience interface (UEI), talking face
module and few-shot text-to-speech (TTS) module. The system firstly clones the
target speaker's voice, and then generates the speech, and finally generate an
avatar with appropriate lip and head movements. Under any scenario, users only
need to replace slides with different notes to generate another new video. The
demo has been released here and will be published as free software for use.

本文提出了一种名为 Pre-Avatar 的系统，通过一张正面照片和 3 分钟的语音记录，生成一个具有说话者形象的演示视频，核心技术包括用户体验界面 (UEI)、说话人形象生成、少样本文本到语音 (TTS) 等几个子模块。用户只需要用不同的注释替换幻灯片，即可生成新视频，这将降低沟通材料的制作和复制成本。

Pre-Avatar：使用漫谈头像的自动演示生成框架

Pre-Avatar: An Automatic Presentation Generation Framework Leveraging Talking Avatar

The presence of a corresponding talking face has been shown to significantly
improve speech intelligibility in noisy conditions and for hearing impaired
population. In this paper, we present a system that can generate landmark
points of a talking face from an acoustic speech in real time. The system uses
a long short-term memory (LSTM) network and is trained on frontal videos of 27
different speakers with automatically extracted face landmarks. After training,
it can produce talking face landmarks from the acoustic speech of unseen
speakers and utterances. The training phase contains three key steps. We first
transform landmarks of the first video frame to pin the two eye points into two
predefined locations and apply the same transformation on all of the following
video frames. We then remove the identity information by transforming the
landmarks into a mean face shape across the entire training dataset. Finally,
we train an LSTM network that takes the first- and second-order temporal
differences of the log-mel spectrogram as input to predict face landmarks in
each frame. We evaluate our system using the mean-squared error (MSE) loss of
landmarks of lips between predicted and ground-truth landmarks as well as their
first- and second-order temporal differences. We further evaluate our system by
conducting subjective tests, where the subjects try to distinguish the real and
fake videos of talking face landmarks. Both tests show promising results.

本文介绍了一个能够从声学语音中实时生成说话人脸关键点的系统，该系统使用一个 LSTM 网络进行训练并能够在不同说话人和话语上进行产生。

从语音生成说话脸部标记

Generating Talking Face Landmarks from Speech

We present a method for generating a video of a talking face. The method
takes as inputs: (i) still images of the target face, and (ii) an audio speech
segment; and outputs a video of the target face lip synched with the audio. The
method runs in real time and is applicable to faces and audio not seen at
training time.
To achieve this we propose an encoder-decoder CNN model that uses a joint
embedding of the face and audio to generate synthesised talking face video
frames. The model is trained on tens of hours of unlabelled videos.
We also show results of re-dubbing videos using speech from a different
person.

本文提出了一种生成说话脸部视频的方法，该方法使用了联合嵌入的编码器解码器 CNN 模型对目标音频和人脸进行建模来生成视频，并且还展示了使用不同人的语音重新配音的结果。