Prompt tuning is a new few-shot transfer learning technique that only tunes
the learnable prompt for pre-trained vision and language models such as CLIP.
However, existing prompt tuning methods tend to learn spurious or entangled
representations, which leads to poor generalization to unseen concepts. Towards
non-spurious and efficient prompt learning from limited examples, this paper
presents a novel \underline{\textbf{C}}ounterfactual
\underline{\textbf{P}}rompt \underline{\textbf{L}}earning (CPL) method for
vision and language models, which simultaneously employs counterfactual
generation and contrastive learning in a joint optimization framework.
Particularly, CPL constructs counterfactual by identifying minimal non-spurious
feature change between semantically-similar positive and negative samples that
causes concept change, and learns more generalizable prompt representation from
both factual and counterfactual examples via contrastive learning. Extensive
experiments demonstrate that CPL can obtain superior few-shot performance on
different vision and language tasks than previous prompt tuning methods on
CLIP. On image classification, we achieve 3.55\% average relative improvement
on unseen classes across seven datasets; on image-text retrieval and visual
question answering, we gain up to 4.09\% and 25.08\% relative improvements
across three few-shot scenarios on unseen test sets respectively.

本文提出了一种新的对视觉和语言模型进行计数事实学习（counterfactual prompt learning，CPL）的方法，该方法同时采用反事实生成和对比学习的方法，在一个联合优化框架中从真实和反事实例子中学习更具有普适性的 prompt 表示。实验测试表明，CPL 在不同的视觉和语言任务上都可以获得优秀的 few-shot 性能。