A true interpreting agent not only understands sign language and translates
to text, but also understands text and translates to signs. Much of the AI work
in sign language translation to date has focused mainly on translating from
signs to text. Towards the latter goal, we propose a text-to-sign translation
model, SignNet, which exploits the notion of similarity (and dissimilarity) of
visual signs in translating. This module presented is only one part of a
dual-learning two task process involving text-to-sign (T2S) as well as
sign-to-text (S2T). We currently implement SignNet as a single channel
architecture so that the output of the T2S task can be fed into S2T in a
continuous dual learning framework. By single channel, we refer to a single
modality, the body pose joints.
In this work, we present SignNet, a T2S task using a novel metric embedding
learning process, to preserve the distances between sign embeddings relative to
their dissimilarity. We also describe how to choose positive and negative
examples of signs for similarity testing. From our analysis, we observe that
metric embedding learning-based model perform significantly better than the
other models with traditional losses, when evaluated using BLEU scores. In the
task of gloss to pose, SignNet performed as well as its state-of-the-art (SoTA)
counterparts and outperformed them in the task of text to pose, by showing
noteworthy enhancements in BLEU 1 - BLEU 4 scores (BLEU 1: 31->39; ~26%
improvement and BLEU 4: 10.43->11.84; ~14\% improvement) when tested on the
popular RWTH PHOENIX-Weather-2014T benchmark dataset

本文介绍了一种基于度量嵌入学习的方法，利用相似性（和不相似性）来进行文本到手语的翻译，并说明了如何选择手语的正负例进行相似性测试。在 RWTH PHOENIX-Weather-2014T 基准数据集上的测试中，使用 SignNet 模型取得了比现有技术更好的表现。

SignNet：利用度量嵌入学习生成单通道符号

SignNet: Single Channel Sign Generation using Metric Embedded Learning

This article presents an original method for Text-to-Sign Translation. It
compensates data scarcity using a domain-specific parallel corpus of alignments
between text and hierarchical formal descriptions of Sign Language videos in
AZee. Based on the detection of similarities present in the source text, the
proposed algorithm recursively exploits matches and substitutions of aligned
segments to build multiple candidate translations for a novel statement. This
helps preserving Sign Language structures as much as possible before falling
back on literal translations too quickly, in a generative way. The resulting
translations are in the form of AZee expressions, designed to be used as input
to avatar synthesis systems. We present a test set tailored to showcase its
potential for expressiveness and generation of idiomatic target language, and
observed limitations. This work finally opens prospects on how to evaluate
translation and linguistic aspects, such as accuracy and grammatical fluency.

本文提出了一种基于 AZee 的文本到手语翻译的算法，利用特定领域的平行语料库来解决数据短缺的问题，通过检测源文本中存在的相似性，递归地利用匹配和对齐段的替换来建立一个句子的多个候选翻译，这有助于以一种生成的方式尽可能保留手语结构。最终的翻译以 AZee 表达式的形式呈现，旨在作为输入到 Avatar 合成系统中。我们提供了一个测试集来展示其表现，并观察到了一些局限性。这项工作最终开启了评估翻译和语言方面的可能性，例如准确性和语法流畅性。