Gloss-free sign language translation (SLT) aims to develop well-performing
SLT systems with no requirement for the costly gloss annotations, but currently
still lags behind gloss-based approaches significantly. In this paper, we
identify a representation density problem that could be a bottleneck in
restricting the performance of gloss-free SLT. Specifically, the representation
density problem describes that the visual representations of semantically
distinct sign gestures tend to be closely packed together in feature space,
which makes gloss-free methods struggle with distinguishing different sign
gestures and suffer from a sharp performance drop. To address the
representation density problem, we introduce a simple but effective contrastive
learning strategy, namely SignCL, which encourages gloss-free models to learn
more discriminative feature representation in a self-supervised manner. Our
experiments demonstrate that the proposed SignCL can significantly reduce the
representation density and improve performance across various translation
frameworks. Specifically, SignCL achieves a significant improvement in BLEU
score for the Sign Language Transformer and GFSLT-VLP on the CSL-Daily dataset
by 39% and 46%, respectively, without any increase of model parameters.
Compared to Sign2GPT, a state-of-the-art method based on large-scale
pre-trained vision and language models, SignCL achieves better performance with
only 35% of its parameters. Implementation and Checkpoints are available at
this https URL

在这篇论文中，我们发现表示密度问题是妨碍无法使用例句注释式手语翻译系统性能的瓶颈，并引入了一种名为 SignCL 的简单但有效的对比学习策略来解决这个问题，通过自我监督的方式改善了无法使用例句注释式方法中特征表示的可区分性。实验证明，SignCL 显著减少了表示密度，并在各种翻译框架上提高了性能，在 CSL-Daily 数据集上，SignCL 使得手语变形器和 GFSLT-VLP 的 BLEU 得分分别提高了 39％和 46％，而模型参数没有增加。与基于大规模预训练视觉和语言模型的最先进方法 Sign2GPT 相比，SignCL 只使用了其参数的 35％并获得了更好的性能。