The increase of web-scale weakly labelled image-text pairs have greatly
facilitated the development of large-scale vision-language models (e.g., CLIP),
which have shown impressive generalization performance over a series of
downstream tasks. However, the massive model size and scarcity of available
data limit their applications to fine-tune the whole model in downstream tasks.
Besides, fully fine-tuning the model easily forgets the generic essential
knowledge acquired in the pretraining stage and overfits the downstream data.
To enable high efficiency when adapting these large vision-language models
(e.g., CLIP) to performing continuous sign language recognition (CSLR) while
preserving their generalizability, we propose a novel strategy (AdaptSign).
Especially, CLIP is adopted as the visual backbone to extract frame-wise
features whose parameters are fixed, and a set of learnable modules are
introduced to model spatial sign variations or capture temporal sign movements.
The introduced additional modules are quite lightweight, only owning 3.2% extra
computations with high efficiency. The generic knowledge acquired in the
pretraining stage is well-preserved in the frozen CLIP backbone in this
process. Extensive experiments show that despite being efficient, AdaptSign is
able to demonstrate superior performance across a series of CSLR benchmarks
including PHOENIX14, PHOENIX14-T, CSL-Daily and CSL compared to existing
methods. Visualizations show that AdaptSign could learn to dynamically pay
major attention to the informative spatial regions and cross-frame trajectories
in sign videos.

通过采用 CLIP 作为视觉主干骨架，在固定参数的同时引入一组可学习模块，AdaptSign 能够高效地适应连续手语识别任务，并在包括 PHOENIX14、PHOENIX14-T、CSL-Daily 和 CSL 等不同基准测试中展现出卓越性能。可视化结果显示，AdaptSign 能够动态地关注手语视频中信息丰富的空间区域和跨帧轨迹。