Large-scale vision-language models (VLMs) like CLIP successfully find
correspondences between images and text. Through the standard deterministic
mapping process, an image or a text sample is mapped to a single vector in the
embedding space. This is problematic: as multiple samples (images or text) can
abstract the same concept in the physical world, deterministic embeddings do
not reflect the inherent ambiguity in the embedding space. We propose ProbVLM,
a probabilistic adapter that estimates probability distributions for the
embeddings of pre-trained VLMs via inter/intra-modal alignment in a post-hoc
manner without needing large-scale datasets or computing. On four challenging
datasets, i.e., COCO, Flickr, CUB, and Oxford-flowers, we estimate the
multi-modal embedding uncertainties for two VLMs, i.e., CLIP and BLIP, quantify
the calibration of embedding uncertainties in retrieval tasks and show that
ProbVLM outperforms other methods. Furthermore, we propose active learning and
model selection as two real-world downstream tasks for VLMs and show that the
estimated uncertainty aids both tasks. Lastly, we present a novel technique for
visualizing the embedding distributions using a large-scale pre-trained latent
diffusion model.

该研究提出了 ProbVLM，一种新的方法来评估大规模视觉语言模型的多模态嵌入不确定性，以及如何利用该不确定性来提高主动学习和模型选择的效果。