Recently, substantial advancements in pre-trained vision-language models have
greatly enhanced the capabilities of multi-modal dialog systems. These models
have demonstrated significant improvements by fine-tuning on downstream tasks.
However, the existing pre-trained models primarily focus on effectively
capturing the alignment between vision and language modalities, often ignoring
the intricate nature of dialog context. In this paper, we propose a
parameter-efficient prompt-tuning method named DialCLIP for multi-modal dialog
retrieval. Specifically, our approach introduces a multi-modal context prompt
generator to learn context features which are subsequently distilled into
prompts within the pre-trained vision-language model CLIP. Besides, we
introduce domain prompt to mitigate the disc repancy from the downstream dialog
data. To facilitate various types of retrieval, we also design multiple experts
to learn mappings from CLIP outputs to multi-modal representation space, with
each expert being responsible to one specific retrieval type. Extensive
experiments show that DialCLIP achieves state-of-the-art performance on two
widely recognized benchmark datasets (i.e., PhotoChat and MMDialog) by tuning a
mere 0.04% of the total parameters. These results highlight the efficacy and
efficiency of our proposed approach, underscoring its potential to advance the
field of multi-modal dialog retrieval.

提出了一种名为 DialCLIP 的参数高效的提示调整方法，用于多模态对话检索，通过仅调整总参数的 0.04％实现了在两个广泛认可的基准数据集上的最新性能，突出了该方法的功效和效率，强调其推动多模态对话检索领域发展的潜力。