Pre-trained vision-language models (VLMs) have shown impressive performance
on various downstream tasks by utilizing knowledge learned from large data. In
general, the performance of VLMs on target tasks can be further improved by
prompt tuning, which adds context to the input image or text. By leveraging
data from target tasks, various prompt-tuning methods have been studied in the
literature. A key to prompt tuning is the feature space alignment between two
modalities via learnable vectors with model parameters fixed. We observed that
the alignment becomes more effective when embeddings of each modality are
`well-arranged' in the latent space. Inspired by this observation, we proposed
distribution-aware prompt tuning (DAPT) for vision-language models, which is
simple yet effective. Specifically, the prompts are learned by maximizing
inter-dispersion, the distance between classes, as well as minimizing the
intra-dispersion measured by the distance between embeddings from the same
class. Our extensive experiments on 11 benchmark datasets demonstrate that our
method significantly improves generalizability. The code is available at
this https URL

通过学习大规模数据的知识，预训练的视觉 - 语言模型（VLMs）在各种下游任务中表现出色。本文提出了分布感知的提示微调（DAPT），通过对齐两种模态之间的特征空间，并对每种模态的嵌入进行良好排列，显著改善了模型的泛化能力。