Automatic Sign Language Translation requires the integration of both computer
vision and natural language processing to effectively bridge the communication
gap between sign and spoken languages. However, the deficiency in large-scale
training data to support sign language translation means we need to leverage
resources from spoken language. We introduce, Sign2GPT, a novel framework for
sign language translation that utilizes large-scale pretrained vision and
language models via lightweight adapters for gloss-free sign language
translation. The lightweight adapters are crucial for sign language
translation, due to the constraints imposed by limited dataset sizes and the
computational requirements when training with long sign videos. We also propose
a novel pretraining strategy that directs our encoder to learn sign
representations from automatically extracted pseudo-glosses without requiring
gloss order information or annotations. We evaluate our approach on two public
benchmark sign language translation datasets, namely RWTH-PHOENIX-Weather 2014T
and CSL-Daily, and improve on state-of-the-art gloss-free translation
performance with a significant margin.

利用大规模预训练视觉和语言模型通过轻量级适配器实现无语言标注的手语翻译的新型框架 Sign2GPT，在两个公共基准手语翻译数据集上评估并取得明显优于现有技术的无语言标注翻译性能提升。

Sign2GPT：利用大型语言模型进行无术语手语翻译

Sign2GPT: Leveraging Large Language Models for Gloss-Free Sign Language  Translation

Automatic Sign Language Translation (SLT) is a research avenue of great
societal impact. End-to-End SLT facilitates the interaction of Hard-of-Hearing
(HoH) with hearing people, thus improving their social life and opportunities
for participation in social life. However, research within this frame of
reference is still in its infancy, and current resources are particularly
limited. Existing SLT methods are either of low translation ability or are
trained and evaluated on datasets of restricted vocabulary and questionable
real-world value. A characteristic example is Phoenix2014T benchmark dataset,
which only covers weather forecasts in German Sign Language. To address this
shortage of resources, we introduce a newly constructed collection of 29653
Greek Sign Language video-translation pairs which is based on the official
syllabus of Greek Elementary School. Our dataset covers a wide range of
subjects. We use this novel dataset to train recent state-of-the-art
Transformer-based methods widely used in SLT research. Our results demonstrate
the potential of our introduced dataset to advance SLT research by offering a
favourable balance between usability and real-world value.

自动手语翻译是一个具有巨大社会影响的研究领域，通过端到端手语翻译可以促进听力有困难的人与听力正常的人之间的交流，从而提高他们的社交生活和参与社会生活的机会。针对现有资源有限的问题，研究者构建了一个新的希腊手语视频翻译数据集，包含 29653 个视频翻译对，基于希腊小学的官方课程，该数据集覆盖了广泛的主题，并使用该数据集训练了最新的基于 Transformer 的方法，结果表明该数据集在提供可用性和实际价值方面具有潜力推动 SLT 研究。

一种新的端到端手语翻译数据集：希腊小学数据集

A New Dataset for End-to-End Sign Language Translation: The Greek  Elementary School Dataset

The advances in automatic sign language translation (SLT) to spoken languages
have been mostly benchmarked with datasets of limited size and restricted
domains. Our work advances the state of the art by providing the first baseline
results on How2Sign, a large and broad dataset.
We train a Transformer over I3D video features, using the reduced BLEU as a
reference metric for validation, instead of the widely used BLEU score. We
report a result of 8.03 on the BLEU score, and publish the first open-source
implementation of its kind to promote further advances.

该研究使用 Transformer 和 I3D 视频特征对 How2Sign 数据集进行训练，并以降低的 BLEU 作为参考指标，获得了 8.03 的 BLEU 得分，提供了第一个开源实现，推进了自动手语翻译技术的发展。