Identifying labels that did not appear during training, known as multi-label
zero-shot learning, is a non-trivial task in computer vision. To this end,
recent studies have attempted to explore the multi-modal knowledge of
vision-language pre-training (VLP) models by knowledge distillation, allowing
to recognize unseen labels in an open-vocabulary manner. However, experimental
evidence shows that knowledge distillation is suboptimal and provides limited
performance gain in unseen label prediction. In this paper, a novel query-based
knowledge sharing paradigm is proposed to explore the multi-modal knowledge
from the pretrained VLP model for open-vocabulary multi-label classification.
Specifically, a set of learnable label-agnostic query tokens is trained to
extract critical vision knowledge from the input image, and further shared
across all labels, allowing them to select tokens of interest as visual clues
for recognition. Besides, we propose an effective prompt pool for robust label
embedding, and reformulate the standard ranking learning into a form of
classification to allow the magnitude of feature vectors for matching, which
both significantly benefit label recognition. Experimental results show that
our framework significantly outperforms state-of-the-art methods on zero-shot
task by 5.9% and 4.5% in mAP on the NUS-WIDE and Open Images, respectively.

提出了一种基于查询的知识共享范式，通过预训练的 VLP 模型从输入图像中提取关键的视觉知识，并通过有效的提示池进行鲁棒的标签嵌入，进而在零样本任务中显著提高了标签识别的性能。