We introduce compositional soft prompting (CSP), a parameter-efficient
learning technique to improve the zero-shot compositionality of large-scale
pretrained vision-language models (VLMs). VLMs can represent arbitrary classes
as natural language prompts in their flexible text encoders, but they
underperform state-of-the-art methods on compositional zero-shot benchmark
tasks. To improve VLMs, we propose a novel form of soft prompting. We treat the
attributes and objects that are composed to define classes as learnable tokens
of vocabulary and tune them on multiple prompt compositions. During inference,
we recompose the learned attribute-object vocabulary in new combinations. We
show that CSP outperforms the original VLM on benchmark datasets by an average
of 10.9 percentage points on AUC. CSP also outperforms CoOp, a soft prompting
method that tunes the prefix context, by an average of 5.8 percentage points on
AUC. We perform additional experiments to show that CSP improves generalization
to attribute-only classification, higher-order attribute-attribute-object
compositions, and combinations of pretrained attributes and fine-tuned objects.

本文提出了一种叫做 CSP 的组成式软提示技术，用于提高大规模预训练视觉 - 语言模型的零样本组成性能；对比已有的方法，CSP 通过训练学习组件而不是固定的预设生词表，可以实现更好的预测表现。

学习生成软提示用于组合式零样本学习

Learning to Compose Soft Prompts for Compositional Zero-Shot Learning

This paper introduces a manually annotated video dataset of unusual actions,
namely RareAct, including actions such as "blend phone", "cut keyboard" and
"microwave shoes". RareAct aims at evaluating the zero-shot and few-shot
compositionality of action recognition models for unlikely compositions of
common action verbs and object nouns. It contains 122 different actions which
were obtained by combining verbs and nouns rarely co-occurring together in the
large-scale textual corpus from HowTo100M, but that frequently appear
separately. We provide benchmarks using a state-of-the-art HowTo100M pretrained
video and text model and show that zero-shot and few-shot compositionality of
actions remains a challenging and unsolved task.

本文介绍了一项手动注释的视频数据集 RareAct，该数据集包含了不同的不寻常的动作，如 “混合手机”、“切割键盘” 和 “微波鞋”，旨在评估行为识别模型的零射击和少射击组合性。该数据集包含 122 种不同的动作，这些动作由 HowTo100M 中很少共同出现，但经常分别出现的动词和名词组成。我们使用最先进的 HowTo100M 预训练视频和文本模型提供基准测试，并表明零射击和少射击组合性仍然是一个具有挑战性且未解决的任务。