Sign Language Translation (SLT) is a challenging task that aims to translate
sign videos into spoken language. Inspired by the strong translation
capabilities of large language models (LLMs) that are trained on extensive
multilingual text corpora, we aim to harness off-the-shelf LLMs to handle SLT.
In this paper, we regularize the sign videos to embody linguistic
characteristics of spoken language, and propose a novel SignLLM framework to
transform sign videos into a language-like representation for improved
readability by off-the-shelf LLMs. SignLLM comprises two key modules: (1) The
Vector-Quantized Visual Sign module converts sign videos into a sequence of
discrete character-level sign tokens, and (2) the Codebook Reconstruction and
Alignment module converts these character-level tokens into word-level sign
representations using an optimal transport formulation. A sign-text alignment
loss further bridges the gap between sign and text tokens, enhancing semantic
compatibility. We achieve state-of-the-art gloss-free results on two
widely-used SLT benchmarks.

使用大型语言模型训练手语翻译任务，提出了 SignLLM 框架，将手语视频转换为类似语言的表示，通过两个关键模块进行转换，同时采用标记对齐损失来增强语义兼容性，在两个广泛使用的手语翻译基准测试中取得了最先进的无注释结果。