We present a generative adversarial network to synthesize 3D pose sequences
of co-speech upper-body gestures with appropriate affective expressions. Our
network consists of two components: a generator to synthesize gestures from a
joint embedding space of features encoded from the input speech and the seed
poses, and a discriminator to distinguish between the synthesized pose
sequences and real 3D pose sequences. We leverage the Mel-frequency cepstral
coefficients and the text transcript computed from the input speech in separate
encoders in our generator to learn the desired sentiments and the associated
affective cues. We design an affective encoder using multi-scale
spatial-temporal graph convolutions to transform 3D pose sequences into latent,
pose-based affective features. We use our affective encoder in both our
generator, where it learns affective features from the seed poses to guide the
gesture synthesis, and our discriminator, where it enforces the synthesized
gestures to contain the appropriate affective expressions. We perform extensive
evaluations on two benchmark datasets for gesture synthesis from the speech,
the TED Gesture Dataset and the GENEA Challenge 2020 Dataset. Compared to the
best baselines, we improve the mean absolute joint error by 10--33%, the mean
acceleration difference by 8--58%, and the Fr\'echet Gesture Distance by
21--34%. We also conduct a user study and observe that compared to the best
current baselines, around 15.28% of participants indicated our synthesized
gestures appear more plausible, and around 16.32% of participants felt the
gestures had more appropriate affective expressions aligned with the speech.

本文介绍了一种基于生成对抗网络的 3D 姿势序列综合方法，通过合适的情感表达来综合共语上半身手势姿势。