A true interpreting agent not only understands sign language and translates
to text, but also understands text and translates to signs. Much of the AI work
in sign language translation to date has focused mainly on translating from
signs to text. Towards the latter goal, we propose a text-to-sign translation
model, SignNet, which exploits the notion of similarity (and dissimilarity) of
visual signs in translating. This module presented is only one part of a
dual-learning two task process involving text-to-sign (T2S) as well as
sign-to-text (S2T). We currently implement SignNet as a single channel
architecture so that the output of the T2S task can be fed into S2T in a
continuous dual learning framework. By single channel, we refer to a single
modality, the body pose joints.
In this work, we present SignNet, a T2S task using a novel metric embedding
learning process, to preserve the distances between sign embeddings relative to
their dissimilarity. We also describe how to choose positive and negative
examples of signs for similarity testing. From our analysis, we observe that
metric embedding learning-based model perform significantly better than the
other models with traditional losses, when evaluated using BLEU scores. In the
task of gloss to pose, SignNet performed as well as its state-of-the-art (SoTA)
counterparts and outperformed them in the task of text to pose, by showing
noteworthy enhancements in BLEU 1 - BLEU 4 scores (BLEU 1: 31->39; ~26%
improvement and BLEU 4: 10.43->11.84; ~14\% improvement) when tested on the
popular RWTH PHOENIX-Weather-2014T benchmark dataset

本文介绍了一种基于度量嵌入学习的方法，利用相似性（和不相似性）来进行文本到手语的翻译，并说明了如何选择手语的正负例进行相似性测试。在 RWTH PHOENIX-Weather-2014T 基准数据集上的测试中，使用 SignNet 模型取得了比现有技术更好的表现。