Motion transfer of talking-head videos involves generating a new video with
the appearance of a subject video and the motion pattern of a driving video.
Current methodologies primarily depend on a limited number of subject images
and 2D representations, thereby neglecting to fully utilize the multi-view
appearance features inherent in the subject video. In this paper, we propose a
novel 3D-aware talking-head video motion transfer network, Head3D, which fully
exploits the subject appearance information by generating a
visually-interpretable 3D canonical head from the 2D subject frames with a
recurrent network. A key component of our approach is a self-supervised 3D head
geometry learning module, designed to predict head poses and depth maps from 2D
subject video frames. This module facilitates the estimation of a 3D head in
canonical space, which can then be transformed to align with driving video
frames. Additionally, we employ an attention-based fusion network to combine
the background and other details from subject frames with the 3D subject head
to produce the synthetic target video. Our extensive experiments on two public
talking-head video datasets demonstrate that Head3D outperforms both 2D and 3D
prior arts in the practical cross-identity setting, with evidence showing it
can be readily adapted to the pose-controllable novel view synthesis task.

本研究提出了一种新颖的 3D 感知说话人视频动作迁移网络 Head3D，通过生成可视化可解释的 3D 标准头部，从 2D 主体帧中充分利用主体外貌信息，以适应驱动视频帧对齐。我们的方法的一个关键组成部分是自我监督的 3D 头部几何学习模块，能够从 2D 主体视频帧预测头部姿势和深度图。此外，我们还采用基于注意力的融合网络，将主体帧的背景和其他细节与 3D 主体头部相结合，生成合成目标视频。我们在两个公共说话人视频数据集上进行了广泛实验，结果表明 Head3D 在实际的跨身份设置中优于 2D 和 3D 先前方法，有证据显示它能够轻松适应受控姿态的新视图合成任务。

3D 感知的讲话头部视频动作转移

3D-Aware Talking-Head Video Motion Transfer

Modern generators render talking-head videos with impressive levels of
photorealism, ushering in new user experiences such as videoconferencing under
constrained bandwidth budgets. Their safe adoption, however, requires a
mechanism to verify if the rendered video is trustworthy. For instance, for
videoconferencing we must identify cases in which a synthetic video portrait
uses the appearance of an individual without their consent. We term this task
avatar fingerprinting. We propose to tackle it by leveraging facial motion
signatures unique to each person. Specifically, we learn an embedding in which
the motion signatures of one identity are grouped together, and pushed away
from those of other identities, regardless of the appearance in the synthetic
video. Avatar fingerprinting algorithms will be critical as talking head
generators become more ubiquitous, and yet no large scale datasets exist for
this new task. Therefore, we contribute a large dataset of people delivering
scripted and improvised short monologues, accompanied by synthetic videos in
which we render videos of one person using the facial appearance of another.
Project page: this https URL

该研究提出了利用每个人独特的面部运动签名学习嵌入的想法来解决合成视频向肖像进行滥用的问题，同时贡献了一个大规模的数据集来解决这个新的 avatar fingerprinting 任务。

合法使用合成说话头像视频的头像指纹识别

Avatar Fingerprinting for Authorized Use of Synthetic Talking-Head  Videos

When people deliver a speech, they naturally move heads, and this rhythmic
head motion conveys prosodic information. However, generating a lip-synced
video while moving head naturally is challenging. While remarkably successful,
existing works either generate still talkingface videos or rely on
landmark/video frames as sparse/dense mapping guidance to generate head
movements, which leads to unrealistic or uncontrollable video synthesis. To
overcome the limitations, we propose a 3D-aware generative network along with a
hybrid embedding module and a non-linear composition module. Through modeling
the head motion and facial expressions1 explicitly, manipulating 3D animation
carefully, and embedding reference images dynamically, our approach achieves
controllable, photo-realistic, and temporally coherent talking-head videos with
natural head movements. Thoughtful experiments on several standard benchmarks
demonstrate that our method achieves significantly better results than the
state-of-the-art methods in both quantitative and qualitative comparisons. The
code is available on this https URL
lelechen63/Talking-head-Generation-with-Rhythmic-Head-Motion.

本文提出一种基于 3D-aware 生成网络、混合嵌入和非线性合成模块的方法，通过显式建模头部运动和面部表情，精心处理 3D 动画以及动态嵌入参考图像，实现了可控、逼真、时序连贯的说话者头像视频，并在多个标准基准测试中表现出优异的结果。