Vision--Language Models (VLMs) have emerged as the dominant approach for
zero-shot recognition, adept at handling diverse scenarios and significant
distribution changes. However, their deployment in risk-sensitive areas
requires a deeper understanding of their uncertainty estimation capabilities, a
relatively uncharted area. In this study, we explore the calibration properties
of VLMs across different architectures, datasets, and training strategies. In
particular, we analyze the uncertainty estimation performance of VLMs when
calibrated in one domain, label set or hierarchy level, and tested in a
different one. Our findings reveal that while VLMs are not inherently
calibrated for uncertainty, temperature scaling significantly and consistently
improves calibration, even across shifts in distribution and changes in label
set. Moreover, VLMs can be calibrated with a very small set of examples.
Through detailed experimentation, we highlight the potential applications and
importance of our insights, aiming for more reliable and effective use of VLMs
in critical, real-world scenarios.

本研究探究了视觉语言模型（VLMs）在不同架构、数据集和训练策略下的校准性能，发现温度缩放显著且一致地改善了校准性能，即使在分布转变和标签集改变的情况下，借此实验结果，我们强调了我们对 VLMs 的理解在关键实际场景中的潜在应用和重要性，旨在更可靠、有效地使用 VLMs。

视觉语言模型校准的关键因素的实证研究

An Empirical Study Into What Matters for Calibrating Vision-Language  Models

We explore calibration properties at various precisions for three
architectures: ShuffleNetv2, GhostNet-VGG, and MobileOne; and two datasets:
CIFAR-100 and PathMNIST. The quality of calibration is observed to track the
quantization quality; it is well-documented that performance worsens with lower
precision, and we observe a similar correlation with poorer calibration. This
becomes especially egregious at 4-bit activation regime. GhostNet-VGG is shown
to be the most robust to overall performance drop at lower precision. We find
that temperature scaling can improve calibration error for quantized networks,
with some caveats. We hope that these preliminary insights can lead to more
opportunities for explainable and reliable EdgeML.

我们探讨了 ShuffleNetv2、GhostNet-VGG 和 MobileOne 三种架构的不同精度下的校准性能，以及 CIFAR-100 和 PathMNIST 两个数据集。发现校准质量与量化质量呈正相关，低精度下性能下降与校准质量下降存在类似关联，尤其在 4 位激活范围。GhostNet-VGG 在低精度下表现最为稳定。发现温度缩放可以改善量化网络的校准误差，但需注意部分细节。希望这些初步的研究洞察能够为可解释可靠的 EdgeML 提供更多机会。