Pre-trained vision-language models (VLMs) have achieved promising success in
many fields, especially with prompt learning paradigm. In this work, we propose
GIP-COL (Graph-Injected Soft Prompting for COmpositional Learning) to better
explore the compositional zero-shot learning (CZSL) ability of VLMs within the
prompt-based learning framework. The soft prompt in GIPCOL is structured and
consists of the prefix learnable vectors, attribute label and object label. In
addition, the attribute and object labels in the soft prompt are designated as
nodes in a compositional graph. The compositional graph is constructed based on
the compositional structure of the objects and attributes extracted from the
training data and consequently feeds the updated concept representation into
the soft prompt to capture this compositional structure for a better prompting
for CZSL. With the new prompting strategy, GIPCOL achieves state-of-the-art AUC
results on all three CZSL benchmarks, including MIT-States, UT-Zappos, and
C-GQA datasets in both closed and open settings compared to previous non-CLIP
as well as CLIP-based methods. We analyze when and why GIPCOL operates well
given the CLIP backbone and its training data limitations, and our findings
shed light on designing more effective prompts for CZSL

我们提出了 GIP-COL（注入图结构的软提示用于组合学习），在提示为基础的学习框架内更好地探索预训练视觉 - 语言模型（VLMs）的组合式零样本学习（CZSL）能力。使用新的提示策略，GIPCOL 在三个 CZSL 基准测试中取得了最新的 AUC 结果，包括 MIT-States、UT-Zappos 和 C-GQA 数据集，相比之前的非 CLIP 和基于 CLIP 方法，在封闭和开放设置下都取得了最新的结果。我们的研究结果具有启发意义，可为 CZSL 的有效提示设计提供指导。

GIPCOL: 图注入软指导的组合式零样本学习

GIPCOL: Graph-Injected Soft Prompting for Compositional Zero-Shot  Learning

Compositional Zero-Shot Learning (CZSL) aims to train models to recognize
novel compositional concepts based on learned concepts such as attribute-object
combinations. One of the challenges is to model attributes interacted with
different objects, e.g., the attribute ``wet" in ``wet apple" and ``wet cat" is
different. As a solution, we provide analysis and argue that attributes are
conditioned on the recognized object and input image and explore learning
conditional attribute embeddings by a proposed attribute learning framework
containing an attribute hyper learner and an attribute base learner. By
encoding conditional attributes, our model enables to generate flexible
attribute embeddings for generalization from seen to unseen compositions.
Experiments on CZSL benchmarks, including the more challenging C-GQA dataset,
demonstrate better performances compared with other state-of-the-art approaches
and validate the importance of learning conditional attributes. Code is
available at this https URL

该研究提出了一种基于条件属性嵌入的学习框架来解决组合零样本学习中的属性建模问题，并在实验中取得了较优的表现。