Generating stylized talking head with diverse head motions is crucial for
achieving natural-looking videos but still remains challenging. Previous works
either adopt a regressive method to capture the speaking style, resulting in a
coarse style that is averaged across all training data, or employ a universal
network to synthesize videos with different styles which causes suboptimal
performance. To address these, we propose a novel dynamic-weight method, namely
Say Anything withAny Style (SAAS), which queries the discrete style
representation via a generative model with a learned style codebook.
Specifically, we develop a multi-task VQ-VAE that incorporates three closely
related tasks to learn a style codebook as a prior for style extraction. This
discrete prior, along with the generative model, enhances the precision and
robustness when extracting the speaking styles of the given style clips. By
utilizing the extracted style, a residual architecture comprising a canonical
branch and style-specific branch is employed to predict the mouth shapes
conditioned on any driving audio while transferring the speaking style from the
source to any desired one. To adapt to different speaking styles, we steer
clear of employing a universal network by exploring an elaborate HyperStyle to
produce the style-specific weights offset for the style branch. Furthermore, we
construct a pose generator and a pose codebook to store the quantized pose
representation, allowing us to sample diverse head motions aligned with the
audio and the extracted style. Experiments demonstrate that our approach
surpasses state-of-theart methods in terms of both lip-synchronization and
stylized expression. Besides, we extend our SAAS to video-driven style editing
field and achieve satisfactory performance.

生成具有多样化头部动作的风格化说话头部对于实现自然的视频至关重要，本文提出了一种使用学习的风格码本通过生成模型来查询离散风格表示的新颖动态权重方法 Say Anything with Any Style（SAAS），并利用提取的风格和残余架构实现在源风格和任意目标风格之间预测嘴部形状的目的。

任何风格的自由发言

Say Anything with Any Style

One-shot talking head generation has no explicit head movement reference,
thus it is difficult to generate talking heads with head motions. Some existing
works only edit the mouth area and generate still talking heads, leading to
unreal talking head performance. Other works construct one-to-one mapping
between audio signal and head motion sequences, introducing ambiguity
correspondences into the mapping since people can behave differently in head
motions when speaking the same content. This unreasonable mapping form fails to
model the diversity and produces either nearly static or even exaggerated head
motions, which are unnatural and strange. Therefore, the one-shot talking head
generation task is actually a one-to-many ill-posed problem and people present
diverse head motions when speaking. Based on the above observation, we propose
OSM-Net, a \textit{one-to-many} one-shot talking head generation network with
natural head motions. OSM-Net constructs a motion space that contains rich and
various clip-level head motion features. Each basis of the space represents a
feature of meaningful head motion in a clip rather than just a frame, thus
providing more coherent and natural motion changes in talking heads. The
driving audio is mapped into the motion space, around which various motion
features can be sampled within a reasonable range to achieve the one-to-many
mapping. Besides, the landmark constraint and time window feature input improve
the accurate expression feature extraction and video generation. Extensive
experiments show that OSM-Net generates more natural realistic head motions
under reasonable one-to-many mapping paradigm compared with other methods.

基于对上述现象的观察，我们提出了一种具有自然头部动作的一对多单拍说话头生成网络 OSM-Net。OSM-Net 构建了一个包含丰富多样的片级头部动作特征的运动空间，从而实现了合理范围内的一对多映射，生成更自然的真实头部运动。

OSM-Net: 单次对多次的自然头部运动的一对多说话人头部合成

OSM-Net: One-to-Many One-shot Talking Head Generation with Spontaneous  Head Motions

In this paper, we propose a novel text-based talking-head video generation
framework that synthesizes high-fidelity facial expressions and head motions in
accordance with contextual sentiments as well as speech rhythm and pauses. To
be specific, our framework consists of a speaker-independent stage and a
speaker-specific stage. In the speaker-independent stage, we design three
parallel networks to generate animation parameters of the mouth, upper face,
and head from texts, separately. In the speaker-specific stage, we present a 3D
face model guided attention network to synthesize videos tailored for different
individuals. It takes the animation parameters as input and exploits an
attention mask to manipulate facial expression changes for the input
individuals. Furthermore, to better establish authentic correspondences between
visual motions (i.e., facial expression changes and head movements) and audios,
we leverage a high-accuracy motion capture dataset instead of relying on long
videos of specific individuals. After attaining the visual and audio
correspondences, we can effectively train our network in an end-to-end fashion.
Extensive experiments on qualitative and quantitative results demonstrate that
our algorithm achieves high-quality photo-realistic talking-head videos
including various facial expressions and head motions according to speech
rhythms and outperforms the state-of-the-art.

本研究提出了一种基于文本的交谈头视频生成框架，能够按照上下文情感及语音节奏和停顿，合成高保真度的面部表情和头部动作。我们的算法通过两个阶段：一个是针对多个人种实现的通用阶段，一个是基于每一个人的具体情况实现的个性阶段。通过广泛的实验验证，我们的这一算法能够生成高质量、逼真的交谈头视频，且在多项指标上均超过目前领先的技术水平。