We show that the representation of one modality can effectively guide the
prompting of another modality for parameter-efficient multimodal fusion.
Specifically, we first encode one modality and use its representation as a
prior to conditionally prompt all frozen layers of the other modality. This is
achieved by disentangling the vanilla prompt vectors into three types of
specialized prompts that adaptively capture global-level and instance-level
features. To better produce the instance-wise prompt, we introduce the mixture
of prompt experts (MoPE) to dynamically route each instance to the most
suitable prompt experts for encoding. We further study a regularization term to
avoid degenerated prompt expert routing. Thanks to our design, our method can
effectively transfer the pretrained knowledge in unimodal encoders for
downstream multimodal tasks. Compared with vanilla prompting, we show that our
MoPE-based conditional prompting is more expressive, thereby scales better with
training data and the total number of prompts. We also demonstrate that our
prompt tuning is architecture-agnostic, thereby offering high modularity.
Extensive experiments over three multimodal datasets demonstrate
state-of-the-art results, matching or surpassing the performance achieved
through fine-tuning, while only necessitating 0.7% of the trainable parameters.
Code will be released: this https URL

利用一种多模态融合方法，通过一个模态的表示来引导另一个模态的参数高效提示，从而将预训练知识有效转移至多模态任务中。