Sign Language Translation (SLT) first uses a Sign Language Recognition (SLR)
system to extract sign language glosses from videos. Then, a translation system
generates spoken language translations from the sign language glosses. This
paper focuses on the translation system and introduces the STMC-Transformer
which improves on the current state-of-the-art by over 5 and 7 BLEU
respectively on gloss-to-text and video-to-text translation of the
PHOENIX-Weather 2014T dataset. On the ASLG-PC12 corpus, we report an increase
of over 16 BLEU.
We also demonstrate the problem in current methods that rely on gloss
supervision. The video-to-text translation of our STMC-Transformer outperforms
translation of GT glosses. This contradicts previous claims that GT gloss
translation acts as an upper bound for SLT performance and reveals that glosses
are an inefficient representation of sign language. For future SLT research, we
therefore suggest an end-to-end training of the recognition and translation
models, or using a different sign language annotation scheme.

本研究提出了 STMC-Transformer 翻译系统，相对当前最先进技术，在 PHOENIX-Weather 2014T 数据集的亮度 - 文本翻译和视频 - 文本翻译方面提高了 5 和 7 BLEU。在 ASLG-PC12 数据集上，也有超过 16 BLEU 的提高。同时，我们证明了当前方法中的问题，即依赖于 gloss 监督会导致 SLT 表现不佳，并揭示了 gloss 是手语的低效表示方法，因此建议未来的 SLT 研究采用端到端的训练方法或使用不同的手语注释方式。