This work dedicates to continuous sign language recognition (CSLR), which is
a weakly supervised task dealing with the recognition of continuous signs from
videos, without any prior knowledge about the temporal boundaries between
consecutive signs. Data scarcity heavily impedes the progress of CSLR. Existing
approaches typically train CSLR models on a monolingual corpus, which is orders
of magnitude smaller than that of speech recognition. In this work, we explore
the feasibility of utilizing multilingual sign language corpora to facilitate
monolingual CSLR. Our work is built upon the observation of cross-lingual
signs, which originate from different sign languages but have similar visual
signals (e.g., hand shape and motion). The underlying idea of our approach is
to identify the cross-lingual signs in one sign language and properly leverage
them as auxiliary training data to improve the recognition capability of
another. To achieve the goal, we first build two sign language dictionaries
containing isolated signs that appear in two datasets. Then we identify the
sign-to-sign mappings between two sign languages via a well-optimized isolated
sign language recognition model. At last, we train a CSLR model on the
combination of the target data with original labels and the auxiliary data with
mapped labels. Experimentally, our approach achieves state-of-the-art
performance on two widely-used CSLR datasets: Phoenix-2014 and Phoenix-2014T.

该研究旨在通过利用多语种手语语料库来促进单一语种的连续手语识别，通过识别跨语种手语并将其作为辅助训练数据，从而提高其识别能力。实验结果表明，该方法在两个广泛使用的连续手语识别数据集上取得了最先进的性能。