The objective of this paper is to develop a functional system for translating
spoken languages into sign languages, referred to as Spoken2Sign translation.
The Spoken2Sign task is orthogonal and complementary to traditional sign
language to spoken language (Sign2Spoken) translation. To enable Spoken2Sign
translation, we present a simple baseline consisting of three steps: 1)
creating a gloss-video dictionary using existing Sign2Spoken benchmarks; 2)
estimating a 3D sign for each sign video in the dictionary; 3) training a
Spoken2Sign model, which is composed of a Text2Gloss translator, a sign
connector, and a rendering module, with the aid of the yielded gloss-3D sign
dictionary. The translation results are then displayed through a sign avatar.
As far as we know, we are the first to present the Spoken2Sign task in an
output format of 3D signs. In addition to its capability of Spoken2Sign
translation, we also demonstrate that two by-products of our approach-3D
keypoint augmentation and multi-view understanding-can assist in keypoint-based
sign language understanding. Code and models will be available at
this https URL

本文旨在开发一种功能性系统，实现将口语翻译成手语的功能，称之为 Spoken2Sign 翻译。我们提出了一个简单的基线，由三个步骤组成：使用现有的 Sign2Spoken 基准创建一个手语视频字典；为字典中的每个手语视频估计一个 3D 手语；借助产生的手语 - 3D 手语字典，训练一个由 Text2Gloss 翻译器、手语连接器和渲染模块组成的 Spoken2Sign 模型。我们是首次将 Spoken2Sign 任务呈现为 3D 手语输出格式，除了能进行 Spoken2Sign 翻译外，我们还证明了我们方法的两个附带产品 ——3D 关键点增强和多视图理解对基于关键点的手语理解的辅助作用。

口语到手语翻译的简单基准模型与 3D 人偶

A Simple Baseline for Spoken Language to Sign Language Translation with  3D Avatars

Hand gesture serves as a crucial role during the expression of sign language.
Current deep learning based methods for sign language understanding (SLU) are
prone to over-fitting due to insufficient sign data resource and suffer limited
interpretability. In this paper, we propose the first self-supervised
pre-trainable SignBERT+ framework with model-aware hand prior incorporated. In
our framework, the hand pose is regarded as a visual token, which is derived
from an off-the-shelf detector. Each visual token is embedded with gesture
state and spatial-temporal position encoding. To take full advantage of current
sign data resource, we first perform self-supervised learning to model its
statistics. To this end, we design multi-level masked modeling strategies
(joint, frame and clip) to mimic common failure detection cases. Jointly with
these masked modeling strategies, we incorporate model-aware hand prior to
better capture hierarchical context over the sequence. After the pre-training,
we carefully design simple yet effective prediction heads for downstream tasks.
To validate the effectiveness of our framework, we perform extensive
experiments on three main SLU tasks, involving isolated and continuous sign
language recognition (SLR), and sign language translation (SLT). Experimental
results demonstrate the effectiveness of our method, achieving new
state-of-the-art performance with a notable gain.

本研究提出一种自监督预训练 SignBERT+ 框架，该框架采用模型可感知的手部先验，并将手势编码为可视令牌，以更好地捕捉序列上的层次性上下文，并在三个主要 SLU 任务上实现了具有显着性能增益的最新性能。