Automatic Cued Speech Recognition (ACSR) provides an intelligent
human-machine interface for visual communications, where the Cued Speech (CS)
system utilizes lip movements and hand gestures to code spoken language for
hearing-impaired people. Previous ACSR approaches often utilize direct feature
concatenation as the main fusion paradigm. However, the asynchronous modalities
i.e., lip, hand shape and hand position) in CS may cause interference for
feature concatenation. To address this challenge, we propose a transformer
based cross-modal mutual learning framework to prompt multi-modal interaction.
Compared with the vanilla self-attention, our model forces modality-specific
information of different modalities to pass through a modality-invariant
codebook, collating linguistic representations for tokens of each modality.
Then the shared linguistic knowledge is used to re-synchronize multi-modal
sequences. Moreover, we establish a novel large-scale multi-speaker CS dataset
for Mandarin Chinese. To our knowledge, this is the first work on ACSR for
Mandarin Chinese. Extensive experiments are conducted for different languages
i.e., Chinese, French, and British English). Results demonstrate that our model
exhibits superior recognition performance to the state-of-the-art by a large
margin.

该研究提出了一种基于变压器的跨模态相互学习框架，用于提高 Cued Speech 系统的自动识别性能，并在中文、法语和英语语言中取得了显着的识别性能提升。

甩手语识别的跨模态相互学习

Cross-Modal Mutual Learning for Cued Speech Recognition

We present Text2Gestures, a transformer-based learning method to
interactively generate emotive full-body gestures for virtual agents aligned
with natural language text inputs. Our method generates emotionally expressive
gestures by utilizing the relevant biomechanical features for body expressions,
also known as affective features. We also consider the intended task
corresponding to the text and the target virtual agents' intended gender and
handedness in our generation pipeline. We train and evaluate our network on the
MPI Emotional Body Expressions Database and observe that our network produces
state-of-the-art performance in generating gestures for virtual agents aligned
with the text for narration or conversation. Our network can generate these
gestures at interactive rates on a commodity GPU. We conduct a web-based user
study and observe that around 91% of participants indicated our generated
gestures to be at least plausible on a five-point Likert Scale. The emotions
perceived by the participants from the gestures are also strongly positively
correlated with the corresponding intended emotions, with a minimum Pearson
coefficient of 0.77 in the valence dimension.

介绍了一种使用基于变压器的学习方法来生成虚拟代理与自然语言输入对应的带有情感表达的全身手势的方法，通过使用生物力学的特征和考虑目标虚拟代理的性别和利手性，该方法能以交互性的速度在图形处理器上产生实时且富有表现力的手势，经过网页用户研究，其生成的手势在趋向性和情感上与预期高度吻合。