In this paper, we introduce the first comprehensive multilingual sign
language dataset named Prompt2Sign, which builds from public data including
American Sign Language (ASL) and seven others. Our dataset transforms a vast
array of videos into a streamlined, model-friendly format, optimized for
training with translation models like seq2seq and text2text. Building on this
new dataset, we propose SignLLM, the first multilingual Sign Language
Production (SLP) model, which includes two novel multilingual SLP modes that
allow for the generation of sign language gestures from input text or prompt.
Both of the modes can use a new loss and a module based on reinforcement
learning, which accelerates the training by enhancing the model's capability to
autonomously sample high-quality data. We present benchmark results of SignLLM,
which demonstrate that our model achieves state-of-the-art performance on SLP
tasks across eight sign languages.

本研究介绍了 Prompt2Sign，第一个全面的多语种手语数据集，此数据集包括美国手语和其他七种手语。我们还提出了 SignLLM，第一个多语种手语生成模型，包括两种新的多语种手语生成模式。基于该数据集，我们展示了 SignLLM 的基准结果，表明我们的模型在八种手语的手语生成任务中达到了最先进的性能。