We propose a generalized method for boosting the generalization ability of
pre-trained vision-language models (VLMs) while fine-tuning on downstream
few-shot tasks. The idea is realized by exploiting out-of-distribution (OOD)
detection to predict whether a sample belongs to a base distribution or a novel
distribution and then using the score generated by a dedicated competition
based scoring function to fuse the zero-shot and few-shot classifier. The fused
classifier is dynamic, which will bias towards the zero-shot classifier if a
sample is more likely from the distribution pre-trained on, leading to improved
base-to-novel generalization ability. Our method is performed only in test
stage, which is applicable to boost existing methods without time-consuming
re-training. Extensive experiments show that even weak distribution detectors
can still improve VLMs' generalization ability. Specifically, with the help of
OOD detectors, the harmonic mean of CoOp and ProGrad increase by 2.6 and 1.5
percentage points over 11 recognition datasets in the base-to-novel setting.

我们提出了一种通用方法，用于在针对下游少样本任务进行精调时提高预训练视觉 - 语言模型 (VLMs) 的泛化能力。该方法利用了超出分布（OOD）检测来预测样本是否属于基本分布或新颖分布，然后使用由专门的竞争性评分函数生成的分数来融合零样本和少样本分类器。融合的分类器是动态的，如果样本更可能来自预先训练的分布，则会偏向于零样本分类器，从而提高基本到新颖的泛化能力。我们的方法仅在测试阶段执行，适用于提升现有方法而无需耗时的重新训练。大量实验证明，即使是弱分布检测器也可以改进 VLMs 的泛化能力。具体来说，在基本到新颖的设置中，在 11 个识别数据集上，借助 OOD 检测器，CoOp 和 ProGrad 的调和平均数分别提高了 2.6 和 1.5 个百分点。