We present a multimodal learning-based method to simultaneously synthesize
co-speech facial expressions and upper-body gestures for digital characters
using RGB video data captured using commodity cameras. Our approach learns from
sparse face landmarks and upper-body joints, estimated directly from video
data, to generate plausible emotive character motions. Given a speech audio
waveform and a token sequence of the speaker's face landmark motion and
body-joint motion computed from a video, our method synthesizes the motion
sequences for the speaker's face landmarks and body joints to match the content
and the affect of the speech. We design a generator consisting of a set of
encoders to transform all the inputs into a multimodal embedding space
capturing their correlations, followed by a pair of decoders to synthesize the
desired face and pose motions. To enhance the plausibility of synthesis, we use
an adversarial discriminator that learns to differentiate between the face and
pose motions computed from the original videos and our synthesized motions
based on their affective expressions. To evaluate our approach, we extend the
TED Gesture Dataset to include view-normalized, co-speech face landmarks in
addition to body gestures. We demonstrate the performance of our method through
thorough quantitative and qualitative experiments on multiple evaluation
metrics and via a user study. We observe that our method results in low
reconstruction error and produces synthesized samples with diverse facial
expressions and body gestures for digital characters.

使用 RGB 视频数据，我们提出了一种基于多模态学习的方法，可以同时合成数字角色的共语言面部表情和上半身动作。我们的方法从视频数据直接估计稀疏的面部关键点和上半身关节，并根据说话者的面部动作和身体关节动作合成合适的情感角色动作。通过对多个评估指标进行全面定量和定性实验证明，我们的方法具有低重建误差，能够为数字角色产生多样化的面部表情和身体动作。

Speech2UnifiedExpressions: 从可负担的输入同步合成言语情感表情、面部表情和身体表情

Speech2UnifiedExpressions: Synchronous Synthesis of Co-Speech Affective  Face and Body Expressions from Affordable Inputs

We present a generative adversarial network to synthesize 3D pose sequences
of co-speech upper-body gestures with appropriate affective expressions. Our
network consists of two components: a generator to synthesize gestures from a
joint embedding space of features encoded from the input speech and the seed
poses, and a discriminator to distinguish between the synthesized pose
sequences and real 3D pose sequences. We leverage the Mel-frequency cepstral
coefficients and the text transcript computed from the input speech in separate
encoders in our generator to learn the desired sentiments and the associated
affective cues. We design an affective encoder using multi-scale
spatial-temporal graph convolutions to transform 3D pose sequences into latent,
pose-based affective features. We use our affective encoder in both our
generator, where it learns affective features from the seed poses to guide the
gesture synthesis, and our discriminator, where it enforces the synthesized
gestures to contain the appropriate affective expressions. We perform extensive
evaluations on two benchmark datasets for gesture synthesis from the speech,
the TED Gesture Dataset and the GENEA Challenge 2020 Dataset. Compared to the
best baselines, we improve the mean absolute joint error by 10--33%, the mean
acceleration difference by 8--58%, and the Fr\'echet Gesture Distance by
21--34%. We also conduct a user study and observe that compared to the best
current baselines, around 15.28% of participants indicated our synthesized
gestures appear more plausible, and around 16.32% of participants felt the
gestures had more appropriate affective expressions aligned with the speech.

本文介绍了一种基于生成对抗网络的 3D 姿势序列综合方法，通过合适的情感表达来综合共语上半身手势姿势。