Multi-label image recognition is a fundamental task in computer vision.
Recently, vision-language models have made notable advancements in this area.
However, previous methods often failed to effectively leverage the rich
knowledge within language models and instead incorporated label semantics into
visual features in a unidirectional manner. In this paper, we propose a
Prompt-driven Visual-Linguistic Representation Learning (PVLR) framework to
better leverage the capabilities of the linguistic modality. In PVLR, we first
introduce a dual-prompting strategy comprising Knowledge-Aware Prompting (KAP)
and Context-Aware Prompting (CAP). KAP utilizes fixed prompts to capture the
intrinsic semantic knowledge and relationships across all labels, while CAP
employs learnable prompts to capture context-aware label semantics and
relationships. Later, we propose an Interaction and Fusion Module (IFM) to
interact and fuse the representations obtained from KAP and CAP. In contrast to
the unidirectional fusion in previous works, we introduce a Dual-Modal
Attention (DMA) that enables bidirectional interaction between textual and
visual features, yielding context-aware label representations and
semantic-related visual representations, which are subsequently used to
calculate similarities and generate final predictions for all labels. Extensive
experiments on three popular datasets including MS-COCO, Pascal VOC 2007, and
NUS-WIDE demonstrate the superiority of PVLR.

在这篇论文中，我们提出了一种 Prompt-driven Visual-Linguistic Representation Learning（PVLR）框架，通过双提示策略和交互融合模块，更好地利用了语言模型的能力，以提高多标签图像识别的性能。实验证明了 PVLR 的卓越性能。