Contrastive Language-Image Pretraining (CLIP) model has exhibited remarkable
efficacy in establishing cross-modal connections between texts and images,
yielding impressive performance across a broad spectrum of downstream
applications through fine-tuning. However, for generalization tasks, the
current fine-tuning methods for CLIP, such as CoOp and CoCoOp, demonstrate
relatively low performance on some fine-grained datasets. We recognize the
underlying reason is that these previous methods only projected global features
into the prompt, neglecting the various visual concepts, such as colors,
shapes, and sizes, which are naturally transferable across domains and play a
crucial role in generalization tasks. To address this issue, in this work, we
propose Concept-Guided Prompt Learning (CPL) for vision-language models.
Specifically, we leverage the well-learned knowledge of CLIP to create a visual
concept cache to enable concept-guided prompting. In order to refine the text
features, we further develop a projector that transforms multi-level visual
features into text features. We observe that this concept-guided prompt
learning approach is able to achieve enhanced consistency between visual and
linguistic modalities. Extensive experimental results demonstrate that our CPL
method significantly improves generalization capabilities compared to the
current state-of-the-art methods.

通过概念引导提示学习的方式，CPL 方法显著提高了通用化性能。

概念引导下的提示学习进行视觉 - 语言模型泛化

Concept-Guided Prompt Learning for Generalization in Vision-Language  Models

Prompt tuning, like CoOp, has recently shown promising vision recognizing and
transfer learning ability on various downstream tasks with the emergence of
large pre-trained vision-language models like CLIP. However, we identify that
existing uni-modal prompt tuning approaches may result in sub-optimal
performance since this uni-modal design breaks the original alignment of
textual and visual representations in the pre-trained model. Inspired by the
nature of pre-trained vision-language models, we aim to achieve completeness in
prompt tuning and propose a novel approach called Multi-modal Deep-symphysis
Prompt Tuning, dubbed as MuDPT, which extends independent multi-modal prompt
tuning by additionally learning a model-agnostic transformative network to
allow deep hierarchical bi-directional prompt fusion. We evaluate the
effectiveness of MuDPT on few-shot vision recognition and out-of-domain
generalization tasks. Compared with the state-of-the-art methods, MuDPT
achieves better recognition and generalization ability with an apparent margin
thanks to synergistic alignment of textual and visual representations. Our code
is available at: this https URL

本文介绍了一种基于多模态深度共生的 Prompt Tuning 方法，通过学习一个模型无关的变换网络，实现了深度的双向自然语言和视觉信息融合，在图像识别和领域外泛化方面表现优异。