Vision-language models (VLMs) have emerged as formidable tools, showing their
strong capability in handling various open-vocabulary tasks in image
recognition, text-driven visual content generation, and visual chatbots, to
name a few. In recent years, considerable efforts and resources have been
devoted to adaptation methods for improving downstream performance of VLMs,
particularly on parameter-efficient fine-tuning methods like prompt learning.
However, a crucial aspect that has been largely overlooked is the confidence
calibration problem in fine-tuned VLMs, which could greatly reduce reliability
when deploying such models in the real world. This paper bridges the gap by
systematically investigating the confidence calibration problem in the context
of prompt learning and reveals that existing calibration methods are
insufficient to address the problem, especially in the open-vocabulary setting.
To solve the problem, we present a simple and effective approach called
Distance-Aware Calibration (DAC), which is based on scaling the temperature
using as guidance the distance between predicted text labels and base classes.
The experiments with 7 distinct prompt learning methods applied across 11
diverse downstream datasets demonstrate the effectiveness of DAC, which
achieves high efficacy without sacrificing the inference speed.

通过调整温度值，根据预测文本标签与基本类之间的距离，我们提出了一种名为距离感知校准（DAC）的简单有效方法，来解决视觉语言模型中存在的置信度校准问题。在 11 个不同的下游数据集上对 7 种不同的提示学习方法进行的实验验证了 DAC 的有效性，并且不影响推理速度。