Generating stylized talking head with diverse head motions is crucial for
achieving natural-looking videos but still remains challenging. Previous works
either adopt a regressive method to capture the speaking style, resulting in a
coarse style that is averaged across all training data, or employ a universal
network to synthesize videos with different styles which causes suboptimal
performance. To address these, we propose a novel dynamic-weight method, namely
Say Anything withAny Style (SAAS), which queries the discrete style
representation via a generative model with a learned style codebook.
Specifically, we develop a multi-task VQ-VAE that incorporates three closely
related tasks to learn a style codebook as a prior for style extraction. This
discrete prior, along with the generative model, enhances the precision and
robustness when extracting the speaking styles of the given style clips. By
utilizing the extracted style, a residual architecture comprising a canonical
branch and style-specific branch is employed to predict the mouth shapes
conditioned on any driving audio while transferring the speaking style from the
source to any desired one. To adapt to different speaking styles, we steer
clear of employing a universal network by exploring an elaborate HyperStyle to
produce the style-specific weights offset for the style branch. Furthermore, we
construct a pose generator and a pose codebook to store the quantized pose
representation, allowing us to sample diverse head motions aligned with the
audio and the extracted style. Experiments demonstrate that our approach
surpasses state-of-theart methods in terms of both lip-synchronization and
stylized expression. Besides, we extend our SAAS to video-driven style editing
field and achieve satisfactory performance.

生成具有多样化头部动作的风格化说话头部对于实现自然的视频至关重要，本文提出了一种使用学习的风格码本通过生成模型来查询离散风格表示的新颖动态权重方法 Say Anything with Any Style（SAAS），并利用提取的风格和残余架构实现在源风格和任意目标风格之间预测嘴部形状的目的。

任何风格的自由发言

Say Anything with Any Style

We present a novel deep-learning based approach to producing animator-centric
speech motion curves that drive a JALI or standard FACS-based production
face-rig, directly from input audio. Our three-stage Long Short-Term Memory
(LSTM) network architecture is motivated by psycho-linguistic insights:
segmenting speech audio into a stream of phonetic-groups is sufficient for
viseme construction; speech styles like mumbling or shouting are strongly
co-related to the motion of facial landmarks; and animator style is encoded in
viseme motion curve profiles. Our contribution is an automatic real-time
lip-synchronization from audio solution that integrates seamlessly into
existing animation pipelines. We evaluate our results by: cross-validation to
ground-truth data; animator critique and edits; visual comparison to recent
deep-learning lip-synchronization solutions; and showing our approach to be
resilient to diversity in speaker and language.

本文提出了一种基于深度学习的新型方法，用于直接从输入音频中产生驱动 JALI 或标准 FACS 制作面部捕捉的动画师中心音频运动曲线。