Sign languages are visual languages which convey information by signers'
handshape, facial expression, body movement, and so forth. Due to the inherent
restriction of combinations of these visual ingredients, there exist a
significant number of visually indistinguishable signs (VISigns) in sign
languages, which limits the recognition capacity of vision neural networks. To
mitigate the problem, we propose the Natural Language-Assisted Sign Language
Recognition (NLA-SLR) framework, which exploits semantic information contained
in glosses (sign labels). First, for VISigns with similar semantic meanings, we
propose language-aware label smoothing by generating soft labels for each
training sign whose smoothing weights are computed from the normalized semantic
similarities among the glosses to ease training. Second, for VISigns with
distinct semantic meanings, we present an inter-modality mixup technique which
blends vision and gloss features to further maximize the separability of
different signs under the supervision of blended labels. Besides, we also
introduce a novel backbone, video-keypoint network, which not only models both
RGB videos and human body keypoints but also derives knowledge from sign videos
of different temporal receptive fields. Empirically, our method achieves
state-of-the-art performance on three widely-adopted benchmarks: MSASL, WLASL,
and NMFs-CSL. Codes are available at this https URL

该论文提出了一种自然语言辅助手语识别 (NLA-SLR) 框架，利用手语词汇中的语义信息来降低手语识别中存在的视觉相似性 (VISigns) 问题。该论文设计了语言辅助标签平滑和交互式混合等技术来提高识别性能，并引入了视频关键点网络作为新型的骨干网络来获得更好的结果。实验结果表明，该方法在三个广泛采用的基准测试数据集上均达到了最先进的性能。