Sign Languages are rich multi-channel languages, requiring articulation of
both manual (hands) and non-manual (face and body) features in a precise,
intricate manner. Sign Language Production (SLP), the automatic translation
from spoken to sign languages, must embody this full sign morphology to be
truly understandable by the Deaf community. Previous work has mainly focused on
manual feature production, with an under-articulated output caused by
regression to the mean.
In this paper, we propose an Adversarial Multi-Channel approach to SLP. We
frame sign production as a minimax game between a transformer-based Generator
and a conditional Discriminator. Our adversarial discriminator evaluates the
realism of sign production conditioned on the source text, pushing the
generator towards a realistic and articulate output. Additionally, we fully
encapsulate sign articulators with the inclusion of non-manual features,
producing facial features and mouthing patterns.
We evaluate on the challenging RWTH-PHOENIX-Weather-2014T (PHOENIX14T)
dataset, and report state-of-the art SLP back-translation performance for
manual production. We set new benchmarks for the production of multi-channel
sign to underpin future research into realistic SLP.

本文提出了利用对抗多通道模型进行手语翻译的方法，在模仿人类手语制作过程中加入非手势要素，如面部特征和口型模式，以提高语言翻译的准确度。我们采用基于转换器的生成器和条件判别器的最小化极值过程，根据来源文本制作符合实际的手语，并在 RWTH-PHOENIX-Weather-2014T (PHOENIX14T) 数据集上进行了评估和表现改进。

多通道手语生成对抗训练

Adversarial Training for Multi-Channel Sign Language Production

Sign language recognition (SLR) is a challenging problem, involving complex
manual features, i.e., hand gestures, and fine-grained non-manual features
(NMFs), i.e., facial expression, mouth shapes, etc. Although manual features
are dominant, non-manual features also play an important role in the expression
of a sign word. Specifically, many sign words convey different meanings due to
non-manual features, even though they share the same hand gestures. This
ambiguity introduces great challenges in the recognition of sign words. To
tackle the above issue, we propose a simple yet effective architecture called
Global-local Enhancement Network (GLE-Net), including two mutually promoted
streams towards different crucial aspects of SLR. Of the two streams, one
captures the global contextual relationship, while the other stream captures
the discriminative fine-grained cues. Moreover, due to the lack of datasets
explicitly focusing on this kind of features, we introduce the first
non-manual-features-aware isolated Chinese sign language dataset~(NMFs-CSL)
with a total vocabulary size of 1,067 sign words in daily life. Extensive
experiments on NMFs-CSL and SLR500 datasets demonstrate the effectiveness of
our method.

提出了一种叫做全局 - 局部增强网络（GLE-Net）的有效架构，包括两个相互促进的流，以应对手势识别中手势和非手势特征的影响，同时推出了第一个非手势特征感知的中文手语数据集（NMFs-CSL）。