This paper proposes a simple transfer learning baseline for sign language
translation. Existing sign language datasets (e.g. PHOENIX-2014T, CSL-Daily)
contain only about 10K-20K pairs of sign videos, gloss annotations and texts,
which are an order of magnitude smaller than typical parallel data for training
spoken language translation models. Data is thus a bottleneck for training
effective sign language translation models. To mitigate this problem, we
propose to progressively pretrain the model from general-domain datasets that
include a large amount of external supervision to within-domain datasets.
Concretely, we pretrain the sign-to-gloss visual network on the general domain
of human actions and the within-domain of a sign-to-gloss dataset, and pretrain
the gloss-to-text translation network on the general domain of a multilingual
corpus and the within-domain of a gloss-to-text corpus. The joint model is
fine-tuned with an additional module named the visual-language mapper that
connects the two networks. This simple baseline surpasses the previous
state-of-the-art results on two sign language translation benchmarks,
demonstrating the effectiveness of transfer learning. With its simplicity and
strong performance, this approach can serve as a solid baseline for future
research. Code and models are available at: this https URL

本文提出了一种简单的基于迁移学习的手语翻译模型，通过预训练模型的视觉网络和语言网络来完成迁移学习，进一步提高了手语翻译模型效果。经实验证明，该模型能够显著优于之前的基于手语翻译结果，为今后的手语翻译研究提供了有效的基准。

简单的多模态迁移学习基准方法用于手语翻译

A Simple Multi-Modality Transfer Learning Baseline for Sign Language  Translation

Sign Language Translation (SLT) first uses a Sign Language Recognition (SLR)
system to extract sign language glosses from videos. Then, a translation system
generates spoken language translations from the sign language glosses. This
paper focuses on the translation system and introduces the STMC-Transformer
which improves on the current state-of-the-art by over 5 and 7 BLEU
respectively on gloss-to-text and video-to-text translation of the
PHOENIX-Weather 2014T dataset. On the ASLG-PC12 corpus, we report an increase
of over 16 BLEU.
We also demonstrate the problem in current methods that rely on gloss
supervision. The video-to-text translation of our STMC-Transformer outperforms
translation of GT glosses. This contradicts previous claims that GT gloss
translation acts as an upper bound for SLT performance and reveals that glosses
are an inefficient representation of sign language. For future SLT research, we
therefore suggest an end-to-end training of the recognition and translation
models, or using a different sign language annotation scheme.

本研究提出了 STMC-Transformer 翻译系统，相对当前最先进技术，在 PHOENIX-Weather 2014T 数据集的亮度 - 文本翻译和视频 - 文本翻译方面提高了 5 和 7 BLEU。在 ASLG-PC12 数据集上，也有超过 16 BLEU 的提高。同时，我们证明了当前方法中的问题，即依赖于 gloss 监督会导致 SLT 表现不佳，并揭示了 gloss 是手语的低效表示方法，因此建议未来的 SLT 研究采用端到端的训练方法或使用不同的手语注释方式。