Few-shot recognition aims to train a classification model with only a few
labeled examples of pre-defined concepts, where annotation can be costly in a
downstream task. In another related research area, zero-shot recognition, which
assumes no access to any downstream-task data, has been greatly advanced by
using pretrained Vision-Language Models (VLMs). In this area,
retrieval-augmented learning (RAL) effectively boosts zero-shot accuracy by
retrieving and learning from external data relevant to downstream concepts.
Motivated by these advancements, our work explores RAL for few-shot
recognition. While seemingly straightforward despite being under-explored in
the literature (till now!), we present novel challenges and opportunities for
applying RAL for few-shot recognition. First, perhaps surprisingly, simply
finetuning the VLM on a large amount of retrieved data barely surpasses
state-of-the-art zero-shot methods due to the imbalanced distribution of
retrieved data and its domain gaps compared to few-shot annotated data. Second,
finetuning a VLM on few-shot examples alone significantly outperforms prior
methods, and finetuning on the mix of retrieved and few-shot data yields even
better results. Third, to mitigate the imbalanced distribution and domain gap
issue, we propose Stage-Wise Augmented fineTuning (SWAT) method, which involves
end-to-end finetuning on mixed data for the first stage and retraining the
classifier solely on the few-shot data in the second stage. Extensive
experiments show that SWAT achieves the best performance on standard benchmark
datasets, resoundingly outperforming prior works by ~10% in accuracy. Code is
available at this https URL

我们的研究探索了检索增强学习（RAL）在少样本识别中的应用。在这项工作中，我们提出了利用 Vision-Language Models（VLMs）进行 finetuning 的两个阶段方法：第一阶段对混合数据进行端到端 finetuning，第二阶段仅使用少样本数据重新训练分类器。实验证明，该方法在标准基准数据集上取得了最佳性能，准确度比现有方法提高了约 10%。

基于逐阶段增强微调的小样本识别

Few-Shot Recognition via Stage-Wise Augmented Finetuning

Training an image captioner without annotated image-sentence pairs has gained
traction in recent years. Previous approaches can be categorized into two
strategies: crawling sentences from mismatching corpora and aligning them with
the given images as pseudo annotations, or pre-training the captioner using
external image-text pairs. However, the aligning setting seems to reach its
performance limit due to the quality problem of pairs, and pre-training
requires significant computational resources. To address these challenges, we
propose a new strategy ``LPM + retrieval-augmented learning" where the prior
knowledge from large pre-trained models (LPMs) is leveraged as supervision, and
a retrieval process is integrated to further reinforce its effectiveness.
Specifically, we introduce Retrieval-augmented Pseudo Sentence Generation
(RaPSG), which adopts an efficient approach to retrieve highly relevant short
region descriptions from the mismatching corpora and use them to generate a
variety of pseudo sentences with distinct representations as well as high
quality via LPMs. In addition, a fluency filter and a CLIP-guided training
objective are further introduced to facilitate model optimization. Experimental
results demonstrate that our method surpasses the SOTA pre-training model
(Flamingo3B) by achieving a CIDEr score of 78.1 (+5.1) while utilizing only
0.3% of its trainable parameters (1.3B VS 33M). Importantly, our approach
eliminates the need of computationally expensive pre-training processes on
external datasets (e.g., the requirement of 312M image-text pairs for
Flamingo3B). We further show that with a simple extension, the generated pseudo
sentences can be deployed as weak supervision to boost the 1% semi-supervised
image caption benchmark up to 93.4 CIDEr score (+8.9) which showcases the
versatility and effectiveness of our approach.

提出了一种新的策略 “基于 LPM 和检索增强学习” 来训练图像说明器，该策略利用大型预训练模型（LPM）的先验知识作为监督，并通过检索过程来进一步增强其有效性，在不使用昂贵的预训练过程的情况下，超越了目前最先进的预训练模型，并通过使用生成的伪句子作为弱监督来提高 1％半监督图像说明基准的性能。