Few-shot recognition aims to train a classification model with only a few
labeled examples of pre-defined concepts, where annotation can be costly in a
downstream task. In another related research area, zero-shot recognition, which
assumes no access to any downstream-task data, has been greatly advanced by
using pretrained Vision-Language Models (VLMs). In this area,
retrieval-augmented learning (RAL) effectively boosts zero-shot accuracy by
retrieving and learning from external data relevant to downstream concepts.
Motivated by these advancements, our work explores RAL for few-shot
recognition. While seemingly straightforward despite being under-explored in
the literature (till now!), we present novel challenges and opportunities for
applying RAL for few-shot recognition. First, perhaps surprisingly, simply
finetuning the VLM on a large amount of retrieved data barely surpasses
state-of-the-art zero-shot methods due to the imbalanced distribution of
retrieved data and its domain gaps compared to few-shot annotated data. Second,
finetuning a VLM on few-shot examples alone significantly outperforms prior
methods, and finetuning on the mix of retrieved and few-shot data yields even
better results. Third, to mitigate the imbalanced distribution and domain gap
issue, we propose Stage-Wise Augmented fineTuning (SWAT) method, which involves
end-to-end finetuning on mixed data for the first stage and retraining the
classifier solely on the few-shot data in the second stage. Extensive
experiments show that SWAT achieves the best performance on standard benchmark
datasets, resoundingly outperforming prior works by ~10% in accuracy. Code is
available at this https URL

我们的研究探索了检索增强学习（RAL）在少样本识别中的应用。在这项工作中，我们提出了利用 Vision-Language Models（VLMs）进行 finetuning 的两个阶段方法：第一阶段对混合数据进行端到端 finetuning，第二阶段仅使用少样本数据重新训练分类器。实验证明，该方法在标准基准数据集上取得了最佳性能，准确度比现有方法提高了约 10%。