Prompt-tuning has demonstrated parameter-efficiency in fusing unimodal
foundation models for multimodal tasks. However, its limited adaptivity and
expressiveness lead to suboptimal performance when compared with other tuning
methods. In this paper, we address this issue by disentangling the vanilla
prompts to adaptively capture dataset-level and instance-level features.
Building upon this disentanglement, we introduce the mixture of prompt experts
(MoPE) technique to enhance expressiveness. MoPE leverages multimodal pairing
priors to route the most effective prompt on a per-instance basis. Compared to
vanilla prompting, our MoPE-based conditional prompting exhibits greater
expressiveness for multimodal fusion, scaling better with the training data and
the overall number of trainable parameters. We also study a regularization term
for expert routing, leading to emergent expert specialization, where different
experts focus on different concepts, enabling interpretable soft prompting.
Extensive experiments across three multimodal datasets demonstrate that our
method achieves state-of-the-art results, matching or even surpassing the
performance of fine-tuning, while requiring only 0.8% of the trainable
parameters. Code will be released: this https URL

通过分解原始提示，基于混合提示专家（MoPE）技术实现了适应性捕获数据集级别和实例级别特征，并借助多模态配对先验在每个实例上选择最有效的提示语，从而提高了多模态融合的表达能力和可扩展性。在专家路由方面引入了正则化项，导致不同专家专注于不同概念，实现了可解释的软提示。通过对三个多模态数据集进行广泛实验，证明了我们的方法在性能上达到了最先进的结果，甚至超过了微调，并且仅需占用可训练参数的 0.8%。