Vision--Language Models (VLMs) have emerged as the dominant approach for zero-shot recognition, adept at handling diverse scenarios and significant distribution changes. However, their deployment in risk-sensitive areas requires a deeper understanding of their uncertainty estimation capabilities, a relatively uncharted area. In this study, we explore the calibration properties of VLMs across different architectures, datasets, and training strategies. In particular, we analyze the uncertainty estimation performance of VLMs when calibrated in one domain, label set or hierarchy level, and tested in a different one. Our findings reveal that while VLMs are not inherently calibrated for uncertainty, temperature scaling significantly and consistently improves calibration, even across shifts in distribution and changes in label set. Moreover, VLMs can be calibrated with a very small set of examples. Through detailed experimentation, we highlight the potential applications and importance of our insights, aiming for more reliable and effective use of VLMs in critical, real-world scenarios.

本研究探究了视觉语言模型（VLMs）在不同架构、数据集和训练策略下的校准性能，发现温度缩放显著且一致地改善了校准性能，即使在分布转变和标签集改变的情况下，借此实验结果，我们强调了我们对VLMs的理解在关键实际场景中的潜在应用和重要性，旨在更可靠、有效地使用VLMs。

视觉语言模型校准的关键因素的实证研究