Recent advances in vision-language models have combined contrastive
approaches with generative methods to achieve state-of-the-art (SOTA) on
downstream inference tasks like zero-shot image classification. However, a
persistent issue of these models for image classification is their
out-of-distribution (OOD) generalization capabilities. We first show that when
an OOD data point is misclassified, the correct class can be typically found in
the Top-K predicted classes. In order to steer the model prediction toward the
correct class within the top predicted classes, we propose the Image-Caption
Encoding (ICE) method, a straightforward approach that directly enforces
consistency between the image-conditioned and caption-conditioned predictions
at evaluation time only. Intuitively, we take advantage of unique properties of
the generated captions to guide our local search for the correct class label
within the Top-K predicted classes. We show that our method can be easily
combined with other SOTA methods to enhance Top-1 OOD accuracies by 0.5% on
average and up to 3% on challenging datasets. Our code:
this https URL

最近的视觉语言模型通过将对比方法与生成方法相结合，在下游推理任务（如零样本图像分类）上取得了最先进的效果。然而，图像分类模型的一个持续问题是它们在未知分布下的泛化能力。我们提出了图像 - 字幕编码（ICE）方法，通过直接在评估时仅强制图像条件和字幕条件的预测一致性来引导模型预测到正确的类别。我们展示了该方法可以与其他最先进方法轻松结合，平均提高 0.5％的一流未知分布准确率，在具有挑战性的数据集上高达 3％。