Few-shot dense retrieval (DR) aims to effectively generalize to novel search
scenarios by learning a few samples. Despite its importance, there is little
study on specialized datasets and standardized evaluation protocols. As a
result, current methods often resort to random sampling from supervised
datasets to create "few-data" setups and employ inconsistent training
strategies during evaluations, which poses a challenge in accurately comparing
recent progress. In this paper, we propose a customized FewDR dataset and a
unified evaluation benchmark. Specifically, FewDR employs class-wise sampling
to establish a standardized "few-shot" setting with finely-defined classes,
reducing variability in multiple sampling rounds. Moreover, the dataset is
disjointed into base and novel classes, allowing DR models to be continuously
trained on ample data from base classes and a few samples in novel classes.
This benchmark eliminates the risk of novel class leakage, providing a reliable
estimation of the DR model's few-shot ability. Our extensive empirical results
reveal that current state-of-the-art DR models still face challenges in the
standard few-shot scene. Our code and data will be open-sourced at
this https URL

本文提出 FewDR 数据集和统一评估基准，并消除了现有方法中出现的一些问题。经实证结果表明，当前最先进的 DR 模型仍然在标准 few-shot 场景中面临挑战。

重新思考稠密检索的少样本能力

Rethinking Dense Retrieval's Few-Shot Ability

Recent years have witnessed the significant advance in dense retrieval (DR)
based on powerful pre-trained language models (PLM). DR models have achieved
excellent performance in several benchmark datasets, while they are shown to be
not as competitive as traditional sparse retrieval models (e.g., BM25) in a
zero-shot retrieval setting. However, in the related literature, there still
lacks a detailed and comprehensive study on zero-shot retrieval. In this paper,
we present the first thorough examination of the zero-shot capability of DR
models. We aim to identify the key factors and analyze how they affect
zero-shot retrieval performance. In particular, we discuss the effect of
several key factors related to source training set, analyze the potential bias
from the target dataset, and review and compare existing zero-shot DR models.
Our findings provide important evidence to better understand and develop
zero-shot DR models.

本文通过第一次全面细致的研究，发现了与源数据集相关的关键因素、目标数据集的潜在偏差和现有零样本 DR 模型，从而提供了关于零样本 DR 模型的更好的理解和发展的重要证据。