Applying large-scale pre-trained visual models like CLIP to few-shot action
recognition tasks can benefit performance and efficiency. Utilizing the
"pre-training, fine-tuning" paradigm makes it possible to avoid training a
network from scratch, which can be time-consuming and resource-intensive.
However, this method has two drawbacks. First, limited labeled samples for
few-shot action recognition necessitate minimizing the number of tunable
parameters to mitigate over-fitting, also leading to inadequate fine-tuning
that increases resource consumption and may disrupt the generalized
representation of models. Second, the video's extra-temporal dimension
challenges few-shot recognition's effective temporal modeling, while
pre-trained visual models are usually image models. This paper proposes a novel
method called Multimodal Adaptation of CLIP (MA-CLIP) to address these issues.
It adapts CLIP for few-shot action recognition by adding lightweight adapters,
which can minimize the number of learnable parameters and enable the model to
transfer across different tasks quickly. The adapters we design can combine
information from video-text multimodal sources for task-oriented spatiotemporal
modeling, which is fast, efficient, and has low training costs. Additionally,
based on the attention mechanism, we design a text-guided prototype
construction module that can fully utilize video-text information to enhance
the representation of video prototypes. Our MA-CLIP is plug-and-play, which can
be used in any different few-shot action recognition temporal alignment metric.

本文提出了一种名为 MA-CLIP 的新方法，旨在通过添加轻量级适配器来适应 CLIP 进行少样本动作识别，从而解决传统方法中的参数调优和时间建模的问题。