We present a multimodal learning-based method to simultaneously synthesize
co-speech facial expressions and upper-body gestures for digital characters
using RGB video data captured using commodity cameras. Our approach learns from
sparse face landmarks and upper-body joints, estimated directly from video
data, to generate plausible emotive character motions. Given a speech audio
waveform and a token sequence of the speaker's face landmark motion and
body-joint motion computed from a video, our method synthesizes the motion
sequences for the speaker's face landmarks and body joints to match the content
and the affect of the speech. We design a generator consisting of a set of
encoders to transform all the inputs into a multimodal embedding space
capturing their correlations, followed by a pair of decoders to synthesize the
desired face and pose motions. To enhance the plausibility of synthesis, we use
an adversarial discriminator that learns to differentiate between the face and
pose motions computed from the original videos and our synthesized motions
based on their affective expressions. To evaluate our approach, we extend the
TED Gesture Dataset to include view-normalized, co-speech face landmarks in
addition to body gestures. We demonstrate the performance of our method through
thorough quantitative and qualitative experiments on multiple evaluation
metrics and via a user study. We observe that our method results in low
reconstruction error and produces synthesized samples with diverse facial
expressions and body gestures for digital characters.

使用 RGB 视频数据，我们提出了一种基于多模态学习的方法，可以同时合成数字角色的共语言面部表情和上半身动作。我们的方法从视频数据直接估计稀疏的面部关键点和上半身关节，并根据说话者的面部动作和身体关节动作合成合适的情感角色动作。通过对多个评估指标进行全面定量和定性实验证明，我们的方法具有低重建误差，能够为数字角色产生多样化的面部表情和身体动作。