Pre-trained vision-language models, e.g., CLIP, working with manually
designed prompts have demonstrated great capacity of transfer learning.
Recently, learnable prompts achieve state-of-the-art performance, which however
are prone to overfit to seen classes, failing to generalize to unseen classes.
In this paper, we propose a Knowledge-Aware Prompt Tuning (KAPT) framework for
vision-language models. Our approach takes inspiration from human intelligence
in which external knowledge is usually incorporated into recognizing novel
categories of objects. Specifically, we design two complementary types of
knowledge-aware prompts for the text encoder to leverage the distinctive
characteristics of category-related external knowledge. The discrete prompt
extracts the key information from descriptions of an object category, and the
learned continuous prompt captures overall contexts. We further design an
adaptation head for the visual encoder to aggregate salient attentive visual
cues, which establishes discriminative and task-aware visual representations.
We conduct extensive experiments on 11 widely-used benchmark datasets and the
results verify the effectiveness in few-shot image classification, especially
in generalizing to unseen categories. Compared with the state-of-the-art CoCoOp
method, KAPT exhibits favorable performance and achieves an absolute gain of
3.22% on new classes and 2.57% in terms of harmonic mean.

通过设计两种类型的知识感知提示，离散提示和学习连续提示，以及视觉编码器的适应头部，实现针对视觉 - 语言模型的知识感知提示调整（KAPT）框架在少样本图像分类中的有效性和对未见类别的泛化能力。与最先进的 CoCoOp 方法相比，KAPT 在新类别中获得了 3.22% 的绝对增益和 2.57% 的调和均值增益。