Prompt tuning, like CoOp, has recently shown promising vision recognizing and
transfer learning ability on various downstream tasks with the emergence of
large pre-trained vision-language models like CLIP. However, we identify that
existing uni-modal prompt tuning approaches may result in sub-optimal
performance since this uni-modal design breaks the original alignment of
textual and visual representations in the pre-trained model. Inspired by the
nature of pre-trained vision-language models, we aim to achieve completeness in
prompt tuning and propose a novel approach called Multi-modal Deep-symphysis
Prompt Tuning, dubbed as MuDPT, which extends independent multi-modal prompt
tuning by additionally learning a model-agnostic transformative network to
allow deep hierarchical bi-directional prompt fusion. We evaluate the
effectiveness of MuDPT on few-shot vision recognition and out-of-domain
generalization tasks. Compared with the state-of-the-art methods, MuDPT
achieves better recognition and generalization ability with an apparent margin
thanks to synergistic alignment of textual and visual representations. Our code
is available at: this https URL

本文介绍了一种基于多模态深度共生的 Prompt Tuning 方法，通过学习一个模型无关的变换网络，实现了深度的双向自然语言和视觉信息融合，在图像识别和领域外泛化方面表现优异。