The Contrastive Language-Image Pre-training (CLIP) has recently shown
remarkable generalization on "zero-shot" training and has applied to many
downstream tasks. We explore the adaptation of CLIP to achieve a more efficient
and generalized action recognition method. We propose that the key lies in
explicitly modeling the motion cues flowing in video frames. To that end, we
design a two-stream motion modeling block to capture motion and spatial
information at the same time. And then, the obtained motion cues are utilized
to drive a dynamic prompts learner to generate motion-aware prompts, which
contain much semantic information concerning human actions. In addition, we
propose a multimodal communication block to achieve a collaborative learning
and further improve the performance. We conduct extensive experiments on
HMDB-51, UCF-101, and Kinetics-400 datasets. Our method outperforms most
existing state-of-the-art methods by a significant margin on "few-shot" and
"zero-shot" training. We also achieve competitive performance on "closed-set"
training with extremely few trainable parameters and additional computational
costs.

通过对视频帧中的运动线索进行建模、利用动态提示学习器生成与人类动作相关的运动感知提示并通过多模态交流模块实现协作学习，我们的方法在少样本学习和零样本学习中具有显著优势，并在少参数和额外计算成本的条件下实现了竞争性性能。