Image-text contrastive models such as CLIP learn transferable and robust
representations for zero-shot transfer to a variety of downstream tasks.
However, to obtain strong downstream performances, prompts need to be carefully
curated, which can be a tedious engineering task. To address the issue of
manual prompt engineering, prompt-tuning is used where a set of contextual
vectors are learned by leveraging information from the training data. Despite
their effectiveness, existing prompt-tuning frameworks often lack
interpretability, thus limiting their ability to understand the compositional
nature of images. In this work, we first identify that incorporating
compositional attributes (e.g., a "green" tree frog) in the design of manual
prompts can significantly enhance image-text alignment scores. Building upon
this observation, we propose a novel and interpretable prompt-tuning method
named IntCoOp, which learns to jointly align attribute-level inductive biases
and class embeddings during prompt-tuning. To assess the effectiveness of our
approach, we evaluate IntCoOp across two representative tasks in a few-shot
learning setup: generalization to novel classes, and unseen domain shifts.
Through extensive experiments across 10 downstream datasets on CLIP, we find
that introducing attribute-level inductive biases leads to superior performance
against state-of-the-art prompt tuning frameworks. Notably, in a 16-shot setup,
IntCoOp improves CoOp by 7.35% in average performance across 10 diverse
datasets.

通过引入属性级归纳偏差和类嵌入来提高图像 - 文本对齐分数的一种新颖的和可解释的提示调优方法。我们通过在 CLIP 上的广泛实验中评估 IntCoOp 来证明其有效性，并发现在 10 个不同领域的下游数据集上，引入属性级归纳偏差能够比现有的提示调优框架获得更好的性能，特别是在 16 张图片的情况下，IntCoOp 提高了 10 个不同数据集的平均表现 7.35%。