Pre-trained vision-language models (VLMs) have achieved promising success in many fields, especially with prompt learning paradigm. In this work, we propose GIP-COL (Graph-Injected Soft Prompting for COmpositional Learning) to better explore the compositional zero-shot learning (CZSL) ability of VLMs within the prompt-based learning framework. The soft prompt in GIPCOL is structured and consists of the prefix learnable vectors, attribute label and object label. In addition, the attribute and object labels in the soft prompt are designated as nodes in a compositional graph. The compositional graph is constructed based on the compositional structure of the objects and attributes extracted from the training data and consequently feeds the updated concept representation into the soft prompt to capture this compositional structure for a better prompting for CZSL. With the new prompting strategy, GIPCOL achieves state-of-the-art AUC results on all three CZSL benchmarks, including MIT-States, UT-Zappos, and C-GQA datasets in both closed and open settings compared to previous non-CLIP as well as CLIP-based methods. We analyze when and why GIPCOL operates well given the CLIP backbone and its training data limitations, and our findings shed light on designing more effective prompts for CZSL

我们提出了GIP-COL（注入图结构的软提示用于组合学习），在提示为基础的学习框架内更好地探索预训练视觉-语言模型（VLMs）的组合式零样本学习（CZSL）能力。使用新的提示策略，GIPCOL在三个CZSL基准测试中取得了最新的AUC结果，包括MIT-States、UT-Zappos和C-GQA数据集，相比之前的非CLIP和基于CLIP方法，在封闭和开放设置下都取得了最新的结果。我们的研究结果具有启发意义，可为CZSL的有效提示设计提供指导。

GIPCOL: 图注入软指导的组合式零样本学习