Cued Speech (CS) is a pure visual coding method used by hearing-impaired
people that combines lip reading with several specific hand shapes to make the
spoken language visible. Automatic CS recognition (ACSR) seeks to transcribe
visual cues of speech into text, which can help hearing-impaired people to
communicate effectively. The visual information of CS contains lip reading and
hand cueing, thus the fusion of them plays an important role in ACSR. However,
most previous fusion methods struggle to capture the global dependency present
in long sequence inputs of multi-modal CS data. As a result, these methods
generally fail to learn the effective cross-modal relationships that contribute
to the fusion. Recently, attention-based transformers have been a prevalent
idea for capturing the global dependency over the long sequence in multi-modal
fusion, but existing multi-modal fusion transformers suffer from both poor
recognition accuracy and inefficient computation for the ACSR task. To address
these problems, we develop a novel computation and parameter efficient
multi-modal fusion transformer by proposing a novel Token-Importance-Aware
Attention mechanism (TIAA), where a token utilization rate (TUR) is formulated
to select the important tokens from the multi-modal streams. More precisely,
TIAA firstly models the modality-specific fine-grained temporal dependencies
over all tokens of each modality, and then learns the efficient cross-modal
interaction for the modality-shared coarse-grained temporal dependencies over
the important tokens of different modalities. Besides, a light-weight gated
hidden projection is designed to control the feature flows of TIAA. The
resulting model, named Economical Cued Speech Fusion Transformer (EcoCued),
achieves state-of-the-art performance on all existing CS datasets, compared
with existing transformer-based fusion methods and ACSR fusion methods.

Cued Speech (CS) 是一种纯视觉编码方法，用于辅助听障人士通过将读唇和手势组合以使口语可见。通过提出一种名为 Economical Cued Speech Fusion Transformer（EcoCued）的计算和参数高效的多模态融合变压器模型，该论文改进了自动 CS 识别（ACSR）的识别准确性和计算效率。