Recent large vision-language models such as CLIP have shown remarkable
out-of-distribution (OOD) detection and generalization performance. However,
their zero-shot in-distribution (ID) accuracy is often limited for downstream
datasets. Recent CLIP-based fine-tuning methods such as prompt learning have
demonstrated significant improvements in ID classification and OOD
generalization where OOD labels are available. Nonetheless, it remains unclear
whether the model is reliable to semantic shifts without OOD labels. In this
paper, we aim to bridge the gap and present a comprehensive study to understand
how fine-tuning impact OOD detection for few-shot downstream tasks. By framing
OOD detection as multi-modal concept matching, we establish a connection
between fine-tuning methods and various OOD scores. Our results suggest that a
proper choice of OOD scores is essential for CLIP-based fine-tuning. In
particular, the maximum concept matching (MCM) score provides a promising
solution consistently. We also show that prompt learning demonstrates the
state-of-the-art OOD detection performance over the zero-shot counterpart.

本文探讨了如何通过多模态概念匹配框架来评估诸如 CLIP 等最新视觉语言模型的 fine-tuning 方法对于少样本下游任务的超出分布检测性能的影响，并发现所选的 ODD 得分方法至关重要，MCM 得分方法表现最佳，prompt learning 在超出分布检测性能上表现最好。