We propose a generalized method for boosting the generalization ability of
pre-trained vision-language models (VLMs) while fine-tuning on downstream
few-shot tasks. The idea is realized by exploiting out-of-distribution (OOD)
detection to predict whether a sample belongs to a base distribution or a novel
distribution and then using the score generated by a dedicated competition
based scoring function to fuse the zero-shot and few-shot classifier. The fused
classifier is dynamic, which will bias towards the zero-shot classifier if a
sample is more likely from the distribution pre-trained on, leading to improved
base-to-novel generalization ability. Our method is performed only in test
stage, which is applicable to boost existing methods without time-consuming
re-training. Extensive experiments show that even weak distribution detectors
can still improve VLMs' generalization ability. Specifically, with the help of
OOD detectors, the harmonic mean of CoOp and ProGrad increase by 2.6 and 1.5
percentage points over 11 recognition datasets in the base-to-novel setting.

我们提出了一种通用方法，用于在针对下游少样本任务进行精调时提高预训练视觉 - 语言模型 (VLMs) 的泛化能力。该方法利用了超出分布（OOD）检测来预测样本是否属于基本分布或新颖分布，然后使用由专门的竞争性评分函数生成的分数来融合零样本和少样本分类器。融合的分类器是动态的，如果样本更可能来自预先训练的分布，则会偏向于零样本分类器，从而提高基本到新颖的泛化能力。我们的方法仅在测试阶段执行，适用于提升现有方法而无需耗时的重新训练。大量实验证明，即使是弱分布检测器也可以改进 VLMs 的泛化能力。具体来说，在基本到新颖的设置中，在 11 个识别数据集上，借助 OOD 检测器，CoOp 和 ProGrad 的调和平均数分别提高了 2.6 和 1.5 个百分点。

弱分布检测器提高了视觉语言提示调整的泛化能力

Weak Distribution Detectors Lead to Stronger Generalizability of  Vision-Language Prompt Tuning

Segment Anything Model (SAM) is a new foundation model that can be used as a
zero-shot object segmentation method with the use of either guide prompts such
as bounding boxes, polygons, or points. Alternatively, additional post
processing steps can be used to identify objects of interest after segmenting
everything in an image. Here we present a method using segment anything
together with a series of post processing steps to segment potato leaves,
called Leaf Only SAM. The advantage of this proposed method is that it does not
require any training data to produce its results so has many applications
across the field of plant phenotyping where there is limited high quality
annotated data available. We compare the performance of Leaf Only SAM to a Mask
R-CNN model which has been fine-tuned on our small novel potato leaf dataset.
On the evaluation dataset, Leaf Only SAM finds an average recall of 63.2 and an
average precision of 60.3, compared to recall of 78.7 and precision of 74.7 for
Mask R-CNN. Leaf Only SAM does not perform better than the fine-tuned Mask
R-CNN model on our data, but the SAM based model does not require any extra
training or annotation of our new dataset. This shows there is potential to use
SAM as a zero-shot classifier with the addition of post processing steps.

Segment Anything Model 是一种新的模型，可以用于零样本目标分割，可以使用边界框、多边形或点等提示进行模型训练，也可以使用后处理步骤，在植物表型学领域具有广泛的应用。在与 Mask R-CNN 模型的比较中，Leaf Only SAM 方法的召回率为 63.2，精确度为 60.3；相比之下，Mask R-CNN 的召回率为 78.7，精确度为 74.7。这表明使用后处理步骤的 SAM 模型可能具有零样本分类器的潜力，而无需训练新的数据集。

Leaf Only SAM: 一种用于零样本自动叶片分割的段落化任何东西流水线

Leaf Only SAM: A Segment Anything Pipeline for Zero-Shot Automated Leaf  Segmentation

Thanks to the large pre-trained vision-language models (VLMs) like CLIP, we
can craft a zero-shot classifier by "prompt", e.g., the confidence score of an
image being "[CLASS]" can be obtained by using the VLM provided similarity
measure between the image and the prompt sentence "a photo of a [CLASS]".
Therefore, prompt shows a great potential for fast adaptation of VLMs to
downstream tasks if we fine-tune the prompt-based similarity measure. However,
we find a common failure that improper fine-tuning may not only undermine the
prompt's inherent prediction for the task-related classes, but also for other
classes in the VLM vocabulary. Existing methods still address this problem by
using traditional anti-overfitting techniques such as early stopping and data
augmentation, which lack a principled solution specific to prompt. We present
Prompt-aligned Gradient, dubbed ProGrad, to prevent prompt tuning from
forgetting the the general knowledge learned from VLMs. In particular, ProGrad
only updates the prompt whose gradient is aligned (or non-conflicting) to the
"general direction", which is represented as the gradient of the KL loss of the
pre-defined prompt prediction. Extensive experiments demonstrate the stronger
few-shot generalization ability of ProGrad over state-of-the-art prompt tuning
methods. Codes are available at this https URL

本研究提出了 Prompt-aligned Gradient 算法（ProGrad）来防止针对 Prompt 的微调忘记从 VLM 学到的一般知识，并通过实验证明其比现有方法具有更强的 few-shot 泛化能力。