Learning generalized representations from limited training samples is crucial
for applying deep neural networks in low-resource scenarios. Recently, methods
based on Contrastive Language-Image Pre-training (CLIP) have exhibited
promising performance in few-shot adaptation tasks. To avoid catastrophic
forgetting and overfitting caused by few-shot fine-tuning, existing works
usually freeze the parameters of CLIP pre-trained on large-scale datasets,
overlooking the possibility that some parameters might not be suitable for
downstream tasks. To this end, we revisit CLIP's visual encoder with a specific
focus on its distinctive attention pooling layer, which performs a spatial
weighted-sum of the dense feature maps. Given that dense feature maps contain
meaningful semantic information, and different semantics hold varying
importance for diverse downstream tasks (such as prioritizing semantics like
ears and eyes in pet classification tasks rather than side mirrors), using the
same weighted-sum operation for dense features across different few-shot tasks
might not be appropriate. Hence, we propose fine-tuning the parameters of the
attention pooling layer during the training process to encourage the model to
focus on task-specific semantics. In the inference process, we perform residual
blending between the features pooled by the fine-tuned and the original
attention pooling layers to incorporate both the few-shot knowledge and the
pre-trained CLIP's prior knowledge. We term this method as Semantic-Aware
FinE-tuning (SAFE). SAFE is effective in enhancing the conventional few-shot
CLIP and is compatible with the existing adapter approach (termed SAFE-A).

通过对关注池化层进行微调，强调任务特定语义信息的注意力分布，以提高普通少样本学习的 CLIP 模型的效果。