In this work, we propose a two-stage sign language production (SLP) paradigm
that first encodes sign language sequences into discrete codes and then
autoregressively generates sign language from text based on the learned
codebook. However, existing vector quantization (VQ) methods are fixed-length
encodings, overlooking the uneven information density in sign language, which
leads to under-encoding of important regions and over-encoding of unimportant
regions. To address this issue, we propose a novel dynamic vector quantization
(DVA-VAE) model that can dynamically adjust the encoding length based on the
information density in sign language to achieve accurate and compact encoding.
Then, a GPT-like model learns to generate code sequences and their
corresponding durations from spoken language text. Extensive experiments
conducted on the PHOENIX14T dataset demonstrate the effectiveness of our
proposed method. To promote sign language research, we propose a new large
German sign language dataset, PHOENIX-News, which contains 486 hours of sign
language videos, audio, and transcription texts.Experimental analysis on
PHOENIX-News shows that the performance of our model can be further improved by
increasing the size of the training data. Our project homepage is
this https URL

本文提出了一种两阶段手语生成 (SLP) 方法，通过首先将手语序列编码为离散编码，然后基于学习到的码本从文本中自回归生成手语。为解决现有向量量化方法中固定长度编码忽视手语中信息密度不均匀的问题，我们提出了一种新颖的动态向量量化 (DVA-VAE) 模型，该模型可以根据手语中的信息密度动态调整编码长度以实现准确而紧凑的编码。随后，一种类似于 GPT 的模型通过学习从口语文本生成编码序列及其对应的持续时间。对 PHOENIX14T 数据集进行了大量实验证明了我们提出的方法的有效性。为了推动手语研究，我们提出了一个新的德语手语大规模数据集 PHOENIX-News，该数据集包含 486 小时的手语视频，音频及转录文本。对 PHOENIX-News 的实验分析表明，增加训练数据的规模可以进一步提高我们模型的性能。项目主页详见此链接。