In this paper, we propose P3D, the human part-wise motion context learning
framework for sign language recognition. Our main contributions lie in two
dimensions: learning the part-wise motion context and employing the pose
ensemble to utilize 2D and 3D pose jointly. First, our empirical observation
implies that part-wise context encoding benefits the performance of sign
language recognition. While previous methods of sign language recognition
learned motion context from the sequence of the entire pose, we argue that such
methods cannot exploit part-specific motion context. In order to utilize
part-wise motion context, we propose the alternating combination of a part-wise
encoding Transformer (PET) and a whole-body encoding Transformer (WET). PET
encodes the motion contexts from a part sequence, while WET merges them into a
unified context. By learning part-wise motion context, our P3D achieves
superior performance on WLASL compared to previous state-of-the-art methods.
Second, our framework is the first to ensemble 2D and 3D poses for sign
language recognition. Since the 3D pose holds rich motion context and depth
information to distinguish the words, our P3D outperformed the previous
state-of-the-art methods employing a pose ensemble.

本文提出了 P3D（人体部分运动上下文学习框架）用于手语识别。我们的主要贡献在于两个方面：学习部分运动上下文和利用 2D 和 3D 姿势联合。

人体部分间 3D 动作上下文学习用于手语识别

Human Part-wise 3D Motion Context Learning for Sign Language Recognition

Vision is often used as a complementary modality for audio speech recognition
(ASR), especially in the noisy environment where performance of solo audio
modality significantly deteriorates. After combining visual modality, ASR is
upgraded to the multi-modality speech recognition (MSR). In this paper, we
propose a two-stage speech recognition model. In the first stage, the target
voice is separated from background noises with help from the corresponding
visual information of lip movements, making the model 'listen' clearly. At the
second stage, the audio modality combines visual modality again to better
understand the speech by a MSR sub-network, further improving the recognition
rate. There are some other key contributions: we introduce a pseudo-3D residual
convolution (P3D)-based visual front-end to extract more discriminative
features; we upgrade the temporal convolution block from 1D ResNet with the
temporal convolutional network (TCN), which is more suitable for the temporal
tasks; the MSR sub-network is built on the top of Element-wise-Attention Gated
Recurrent Unit (EleAtt-GRU), which is more effective than Transformer in long
sequences. We conducted extensive experiments on the LRS3-TED and the LRW
datasets. Our two-stage model (audio enhanced multi-modality speech
recognition, AE-MSR) consistently achieves the state-of-the-art performance by
a significant margin, which demonstrates the necessity and effectiveness of
AE-MSR.

本文提出了一种基于视听的两阶段语音识别模型，该模型利用视唇运动信息清晰地区分出背景噪音并提升语音识别率，同时用 P3D 和 EleAtt-GRU 技术进一步提高模型性能，实验证明该模型在 LRS3-TED 和 LRW 数据集上均取得了较大的性能提升，表明 AE-MSR 的必要性和有效性。