Sign language recognition (SLR) is a weakly supervised task that annotates
sign videos as textual glosses. Recent studies show that insufficient training
caused by the lack of large-scale available sign datasets becomes the main
bottleneck for SLR. Most SLR works thereby adopt pretrained visual modules and
develop two mainstream solutions. The multi-stream architectures extend
multi-cue visual features, yielding the current SOTA performances but requiring
complex designs and might introduce potential noise. Alternatively, the
advanced single-cue SLR frameworks using explicit cross-modal alignment between
visual and textual modalities are simple and effective, potentially competitive
with the multi-cue framework. In this work, we propose a novel contrastive
visual-textual transformation for SLR, CVT-SLR, to fully explore the pretrained
knowledge of both the visual and language modalities. Based on the single-cue
cross-modal alignment framework, we propose a variational autoencoder (VAE) for
pretrained contextual knowledge while introducing the complete pretrained
language module. The VAE implicitly aligns visual and textual modalities while
benefiting from pretrained contextual knowledge as the traditional contextual
module. Meanwhile, a contrastive cross-modal alignment algorithm is designed to
explicitly enhance the consistency constraints. Extensive experiments on public
datasets (PHOENIX-2014 and PHOENIX-2014T) demonstrate that our proposed CVT-SLR
consistently outperforms existing single-cue methods and even outperforms SOTA
multi-cue methods.

本文提出了一种新的对比视觉 - 文本转换（CVT-SLR）的方案，以充分探索视觉和语言模态的预训练知识，改进手语识别（SLR）的效果，实验结果表明其优于现有的单模态方法甚至优于 SOTA 多模态方法。