In this work, we tackle the challenge of enhancing the realism and
expressiveness in talking head video generation by focusing on the dynamic and
nuanced relationship between audio cues and facial movements. We identify the
limitations of traditional techniques that often fail to capture the full
spectrum of human expressions and the uniqueness of individual facial styles.
To address these issues, we propose EMO, a novel framework that utilizes a
direct audio-to-video synthesis approach, bypassing the need for intermediate
3D models or facial landmarks. Our method ensures seamless frame transitions
and consistent identity preservation throughout the video, resulting in highly
expressive and lifelike animations. Experimental results demonsrate that EMO is
able to produce not only convincing speaking videos but also singing videos in
various styles, significantly outperforming existing state-of-the-art
methodologies in terms of expressiveness and realism.

在这项工作中，我们通过关注音频线索和面部动作之间的动态和微妙关系，解决了提高说话视频生成中的真实感和表现力的挑战。我们确定了传统技术的局限性，常常无法捕捉到人类表情的全谱和个体面部风格的独特性。为了解决这些问题，我们提出了 EMO，这是一个新颖的框架，利用直接的音频到视频合成方法，绕过了中间的 3D 模型或面部标记的需求。我们的方法确保了平滑的帧过渡和在整个视频中的一致身份保护，从而产生高度表现力和逼真的动画。实验结果表明，EMO 不仅能够产生令人信服的说话视频，还能以各种风格生成唱歌视频，其在表现力和逼真度方面明显优于现有最先进的方法学。

EMO: 表情肖像的生命之躍 - 在弱條件下使用 Audio2Video 擴散模型生成有表情的肖像視頻

EMO: Emote Portrait Alive - Generating Expressive Portrait Videos with  Audio2Video Diffusion Model under Weak Conditions

Talking head video generation aims to animate a human face in a still image
with dynamic poses and expressions using motion information derived from a
target-driving video, while maintaining the person's identity in the source
image. However, dramatic and complex motions in the driving video cause
ambiguous generation, because the still source image cannot provide sufficient
appearance information for occluded regions or delicate expression variations,
which produces severe artifacts and significantly degrades the generation
quality. To tackle this problem, we propose to learn a global facial
representation space, and design a novel implicit identity representation
conditioned memory compensation network, coined as MCNet, for high-fidelity
talking head generation.~Specifically, we devise a network module to learn a
unified spatial facial meta-memory bank from all training samples, which can
provide rich facial structure and appearance priors to compensate warped source
facial features for the generation. Furthermore, we propose an effective query
mechanism based on implicit identity representations learned from the discrete
keypoints of the source image. It can greatly facilitate the retrieval of more
correlated information from the memory bank for the compensation. Extensive
experiments demonstrate that MCNet can learn representative and complementary
facial memory, and can clearly outperform previous state-of-the-art talking
head generation methods on VoxCeleb1 and CelebV datasets. Please check our
\href{this https URL}{Project}.

通过学习全局面部表达空间并设计了一个新颖的隐式身份表示条件化内存补偿网络（MCNet），我们提出了一种高保真的谈话头生成方法，能够从训练样本中学习统一的空间面部元记忆库，并通过离散关键点的隐式身份表示来有效查询这个记忆库以获得更相关的信息进行补偿，实验证明 MCNet 可以学习到代表性和互补的面部记忆，并在 VoxCeleb1 和 CelebV 数据集上明显优于现有的谈话头生成方法。

隐式身份表达条件化记忆补偿网络用于说头视频生成

Implicit Identity Representation Conditioned Memory Compensation Network  for Talking Head video Generation

Talking head video generation aims to produce a synthetic human face video
that contains the identity and pose information respectively from a given
source image and a driving video.Existing works for this task heavily rely on
2D representations (e.g. appearance and motion) learned from the input images.
However, dense 3D facial geometry (e.g. pixel-wise depth) is extremely
important for this task as it is particularly beneficial for us to essentially
generate accurate 3D face structures and distinguish noisy information from the
possibly cluttered background. Nevertheless, dense 3D geometry annotations are
prohibitively costly for videos and are typically not available for this video
generation task. In this paper, we first introduce a self-supervised geometry
learning method to automatically recover the dense 3D geometry (i.e.depth) from
the face videos without the requirement of any expensive 3D annotation data.
Based on the learned dense depth maps, we further propose to leverage them to
estimate sparse facial keypoints that capture the critical movement of the
human head. In a more dense way, the depth is also utilized to learn 3D-aware
cross-modal (i.e. appearance and depth) attention to guide the generation of
motion fields for warping source image representations. All these contributions
compose a novel depth-aware generative adversarial network (DaGAN) for talking
head generation. Extensive experiments conducted demonstrate that our proposed
method can generate highly realistic faces, and achieve significant results on
the unseen human faces.

本文提出了一种基于深度学习的方法，使用自监督几何学习从人脸视频中自动恢复精细的三维几何结构，并应用于生成高度逼真的人脸视频。