Automatic Cued Speech Recognition (ACSR) provides an intelligent
human-machine interface for visual communications, where the Cued Speech (CS)
system utilizes lip movements and hand gestures to code spoken language for
hearing-impaired people. Previous ACSR approaches often utilize direct feature
concatenation as the main fusion paradigm. However, the asynchronous modalities
i.e., lip, hand shape and hand position) in CS may cause interference for
feature concatenation. To address this challenge, we propose a transformer
based cross-modal mutual learning framework to prompt multi-modal interaction.
Compared with the vanilla self-attention, our model forces modality-specific
information of different modalities to pass through a modality-invariant
codebook, collating linguistic representations for tokens of each modality.
Then the shared linguistic knowledge is used to re-synchronize multi-modal
sequences. Moreover, we establish a novel large-scale multi-speaker CS dataset
for Mandarin Chinese. To our knowledge, this is the first work on ACSR for
Mandarin Chinese. Extensive experiments are conducted for different languages
i.e., Chinese, French, and British English). Results demonstrate that our model
exhibits superior recognition performance to the state-of-the-art by a large
margin.

该研究提出了一种基于变压器的跨模态相互学习框架，用于提高 Cued Speech 系统的自动识别性能，并在中文、法语和英语语言中取得了显着的识别性能提升。