We address the task of American Sign Language fingerspelling translation
using videos in the wild. We exploit advances in more accurate hand pose
estimation and propose a novel architecture that leverages the transformer
based encoder-decoder model enabling seamless contextual word translation. The
translation model is augmented by a novel loss term that accurately predicts
the length of the finger-spelled word, benefiting both training and inference.
We also propose a novel two-stage inference approach that re-ranks the
hypotheses using the language model capabilities of the decoder. Through
extensive experiments, we demonstrate that our proposed method outperforms the
state-of-the-art models on ChicagoFSWild and ChicagoFSWild+ achieving more than
10% relative improvement in performance. Our findings highlight the
effectiveness of our approach and its potential to advance fingerspelling
recognition in sign language translation. Code is also available at
this https URL

我们在野外使用视频来处理美国手语拼写翻译的任务。我们利用更精确的手部姿势估计和提出了一种新的架构，它利用基于 Transformer 的编码 - 解码模型，实现了无缝上下文单词翻译。翻译模型通过一种新的损失项，准确预测指拼单词的长度，从而有益于训练和推理。我们还提出了一种新的两阶段推理方法，在解码器的语言模型能力下重新对假设进行排名。通过广泛的实验证明，我们的方法在 ChicagoFSWild 和 ChicagoFSWild + 上优于现有技术模型，相对性能提高了 10% 以上。我们的发现突出了我们方法的有效性和推进手语翻译中的拼写识别的潜力。代码也可在此 https 网址找到。