Many continuous sign language recognition (CSLR) studies adopt
transformer-based architectures for sequence modeling due to their powerful
capacity for capturing global contexts. Nevertheless, vanilla self-attention,
which serves as the core module of the transformer, calculates a weighted
average over all time steps; therefore, the local temporal semantics of sign
videos may not be fully exploited. In this study, we introduce a novel module
in sign language recognition studies, called intra-inter gloss attention
module, to leverage the relationships among frames within glosses and the
semantic and grammatical dependencies between glosses in the video. In the
intra-gloss attention module, the video is divided into equally sized chunks
and a self-attention mechanism is applied within each chunk. This localized
self-attention significantly reduces complexity and eliminates noise introduced
by considering non-relative frames. In the inter-gloss attention module, we
first aggregate the chunk-level features within each gloss chunk by average
pooling along the temporal dimension. Subsequently, multi-head self-attention
is applied to all chunk-level features. Given the non-significance of the
signer-environment interaction, we utilize segmentation to remove the
background of the videos. This enables the proposed model to direct its focus
toward the signer. Experimental results on the PHOENIX-2014 benchmark dataset
demonstrate that our method can effectively extract sign language features in
an end-to-end manner without any prior knowledge, improve the accuracy of CSLR,
and achieve the word error rate (WER) of 20.4 on the test set which is a
competitive result compare to the state-of-the-art which uses additional
supervisions.

本研究介绍了在连续手语识别研究中引入的一种新模块，称为 “内 - 间词汇关注模块”，以利用手语视频中词汇内部帧之间的关系，以及词汇之间的语义和语法依赖关系。在实验中，使用 PHOENIX-2014 基准数据集对所提出的模型进行评估，结果表明该方法可以在无先验知识的情况下有效地提取手语特征，改善连续手语识别的准确性，并在测试集上取得 20.4 的字词错误率（WER），这是与使用额外监督的先进技术相比具有竞争力的结果。