Supervised and self-supervised learning are two main training paradigms for
skeleton-based human action recognition. However, the former one-hot
classification requires labor-intensive predefined action categories
annotations, while the latter involves skeleton transformations (e.g.,
cropping) in the pretext tasks that may impair the skeleton structure. To
address these challenges, we introduce a novel skeleton-based training
framework (C$^2$VL) based on Cross-modal Contrastive learning that uses the
progressive distillation to learn task-agnostic human skeleton action
representation from the Vision-Language knowledge prompts. Specifically, we
establish the vision-language action concept space through vision-language
knowledge prompts generated by pre-trained large multimodal models (LMMs),
which enrich the fine-grained details that the skeleton action space lacks.
Moreover, we propose the intra-modal self-similarity and inter-modal
cross-consistency softened targets in the cross-modal contrastive process to
progressively control and guide the degree of pulling vision-language knowledge
prompts and corresponding skeletons closer. These soft instance discrimination
and self-knowledge distillation strategies contribute to the learning of better
skeleton-based action representations from the noisy skeleton-vision-language
pairs. During the inference phase, our method requires only the skeleton data
as the input for action recognition and no longer for vision-language prompts.
Extensive experiments show that our method achieves state-of-the-art results on
NTU RGB+D 60, NTU RGB+D 120, and PKU-MMD datasets. The code will be available
in the future.

基于骨骼的动作识别的训练方法之一是基于监督学习的独热分类，这需要大量的预定义动作类别注释；而基于自监督学习的方法涉及预处理任务中的骨架变换，可能会损害骨架结构。为了解决这些挑战，我们引入了一种新颖的基于骨骼的训练框架（C$^2$VL），基于跨模态对比学习，使用渐进蒸馏从视觉 - 语言知识提示中学习无关任务的人类骨架动作表征。