End-to-end sign language translation (SLT) aims to convert sign language
videos into spoken language texts directly without intermediate
representations. It has been a challenging task due to the modality gap between
sign videos and texts and the data scarcity of labeled data. To tackle these
challenges, we propose a novel Cross-modality Data Augmentation (XmDA)
framework to transfer the powerful gloss-to-text translation capabilities to
end-to-end sign language translation (i.e. video-to-text) by exploiting pseudo
gloss-text pairs from the sign gloss translation model. Specifically, XmDA
consists of two key components, namely, cross-modality mix-up and
cross-modality knowledge distillation. The former explicitly encourages the
alignment between sign video features and gloss embeddings to bridge the
modality gap. The latter utilizes the generation knowledge from gloss-to-text
teacher models to guide the spoken language text generation. Experimental
results on two widely used SLT datasets, i.e., PHOENIX-2014T and CSL-Daily,
demonstrate that the proposed XmDA framework significantly and consistently
outperforms the baseline models. Extensive analyses confirm our claim that XmDA
enhances spoken language text generation by reducing the representation
distance between videos and texts, as well as improving the processing of
low-frequency words and long sentences.

提出一种名为交叉模态数据增强（XmDA）的框架，以通过从标记数据中利用伪 Gloss-Text 对实现从手语视频到文本的端到端手语翻译

面向端到端手语翻译的跨模态数据增强

Cross-modality Data Augmentation for End-to-End Sign Language  Translation

Sign language translation (SLT) is often decomposed into video-to-gloss
recognition and gloss-to-text translation, where a gloss is a sequence of
transcribed spoken-language words in the order in which they are signed. We
focus here on gloss-to-text translation, which we treat as a low-resource
neural machine translation (NMT) problem. However, unlike traditional
low-resource NMT, gloss-to-text translation differs because gloss-text pairs
often have a higher lexical overlap and lower syntactic overlap than pairs of
spoken languages. We exploit this lexical overlap and handle syntactic
divergence by proposing two rule-based heuristics that generate pseudo-parallel
gloss-text pairs from monolingual spoken language text. By pre-training on the
thus obtained synthetic data, we improve translation from American Sign
Language (ASL) to English and German Sign Language (DGS) to German by up to
3.14 and 2.20 BLEU, respectively.

该研究提出了两个基于规则的启发式方法，从而提高了手语到文本翻译的机器翻译准确性，并成功地将美国手语（ASL）翻译成英语和德国手语（DGS）翻译成德语。