Recent advancements in multimodal foundation models (e.g., CLIP) have
excelled in zero-shot generalization. Prompt tuning involved in the knowledge
transfer from foundation models to downstream tasks has gained significant
attention recently. Existing prompt-tuning methods in cross-modal learning,
however, either solely focus on language branch, or learn vision-language
interaction in a shallow mechanism. In this context, we propose a Deeply
coupled Cross-modal Prompt learning (DCP) method based on CLIP. DCP flexibly
accommodates the interplay between vision and language with a Cross-Modal
Prompt Attention (CMPA) mechanism, which enables the mutual exchange of
respective representation through a well-connected multi-head attention module
progressively and strongly. We then conduct comprehensive few-shot learning
experiments on 11 image classification datasets and analyze the robustness to
domain shift as well. Thorough experimental analysis evidently demonstrates the
superb few-shot generalization and compelling domain adaption capacity of a
well-executed DCP. The code can be found at
\href{https://github.com/GingL/CMPA}{this https URL}.

本文提出了一种基于 CLIP 的深度交互式跨模态提示学习（DCP）方法，可实现视觉和语言之间的灵活相互作用，通过良好连接的多头注意力模块逐步强且稳定地交换各自的表示，并在 11 个图像分类数据集上进行综合少样本学习实验和域自适应能力分析，实验结果显示 DCP 表现出了超凡的少样本泛化性能和具有吸引力的域适应能力。