Automatic gesture synthesis from speech is a topic that has attracted
researchers for applications in remote communication, video games and
Metaverse. Learning the mapping between speech and 3D full-body gestures is
difficult due to the stochastic nature of the problem and the lack of a rich
cross-modal dataset that is needed for training. In this paper, we propose a
novel transformer-based framework for automatic 3D body gesture synthesis from
speech. To learn the stochastic nature of the body gesture during speech, we
propose a variational transformer to effectively model a probabilistic
distribution over gestures, which can produce diverse gestures during
inference. Furthermore, we introduce a mode positional embedding layer to
capture the different motion speeds in different speaking modes. To cope with
the scarcity of data, we design an intra-modal pre-training scheme that can
learn the complex mapping between the speech and the 3D gesture from a limited
amount of data. Our system is trained with either the Trinity speech-gesture
dataset or the Talking With Hands 16.2M dataset. The results show that our
system can produce more realistic, appropriate, and diverse body gestures
compared to existing state-of-the-art approaches.

提出了一种基于变分变压器的新型自动化从语音中合成的三维体手势的框架，通过学习语音和三维手势之间的映射，使用概率分布生成多样化的手势，并通过预训练方案解决数据稀缺性问题，结果显示比现有方法产生更逼真、更恰当和更多样化的身体手势。

BodyFormer：基于 Transformer 的语义引导的三维身体手势合成

BodyFormer: Semantics-guided 3D Body Gesture Synthesis with Transformer

Despite the great promise of Transformers in many sequence modeling tasks
(e.g., machine translation), their deterministic nature hinders them from
generalizing to high entropy tasks such as dialogue response generation.
Previous work proposes to capture the variability of dialogue responses with a
recurrent neural network (RNN)-based conditional variational autoencoder
(CVAE). However, the autoregressive computation of the RNN limits the training
efficiency. Therefore, we propose the Variational Transformer (VT), a
variational self-attentive feed-forward sequence model. The VT combines the
parallelizability and global receptive field of the Transformer with the
variational nature of the CVAE by incorporating stochastic latent variables
into Transformers. We explore two types of the VT: 1) modeling the
discourse-level diversity with a global latent variable; and 2) augmenting the
Transformer decoder with a sequence of fine-grained latent variables. Then, the
proposed models are evaluated on three conversational datasets with both
automatic metric and human evaluation. The experimental results show that our
models improve standard Transformers and other baselines in terms of diversity,
semantic relevance, and human judgment.

本文提出了变分 Transformer（VT）算法，将变分自注意力前馈序列模型与 CVAE 的变分性质相结合，从而有效处理对话响应生成领域中高熵任务的问题，并在多个对话数据集上获得了性能提升。