Motivated by the success of unsupervised neural machine translation (UNMT),
we introduce an unsupervised sign language translation and generation network
(USLNet), which learns from abundant single-modality (text and video) data
without parallel sign language data. USLNet comprises two main components:
single-modality reconstruction modules (text and video) that rebuild the input
from its noisy version in the same modality and cross-modality back-translation
modules (text-video-text and video-text-video) that reconstruct the input from
its noisy version in the different modality using back-translation
procedure.Unlike the single-modality back-translation procedure in text-based
UNMT, USLNet faces the cross-modality discrepancy in feature representation, in
which the length and the feature dimension mismatch between text and video
sequences. We propose a sliding window method to address the issues of aligning
variable-length text with video sequences. To our knowledge, USLNet is the
first unsupervised sign language translation and generation model capable of
generating both natural language text and sign language video in a unified
manner. Experimental results on the BBC-Oxford Sign Language dataset (BOBSL)
and Open-Domain American Sign Language dataset (OpenASL) reveal that USLNet
achieves competitive results compared to supervised baseline models, indicating
its effectiveness in sign language translation and generation.

通过无监督的神经机器翻译 (UNMT) 的成功，我们引入了一种无监督的手语翻译和生成网络 (USLNet)，它能够从丰富的单模态（文本和视频）数据中学习，而不需要平行手语数据。USLNet 由两个主要组成部分组成：单模态重构模块（文本和视频），用于从相同模态的噪声版本重建输入；跨模态回译模块（文本 - 视频 - 文本和视频 - 文本 - 视频），其通过回译过程在不同模态下重建输入。与基于文本的 UNMT 中单模态回译过程不同，USLNet 面临着特征表示中的跨模态差异，其中文本和视频序列之间的长度和特征维度不匹配。我们提出了一种滑动窗口方法来解决文本与视频序列对齐的问题。据我们所知，USLNet 是首个能够以统一方式生成自然语言文本和手语视频的无监督手语翻译和生成模型。在 BBC-Oxford 手语数据集（BOBSL）和开放领域美国手语数据集 (OpenASL) 上的实验结果表明，USLNet 与有监督基线模型相比取得了竞争性的结果，表明其在手语翻译和生成方面的有效性。