Language models, especially pre-trained large language models, have showcased
remarkable abilities as few-shot in-context learners (ICL), adept at adapting
to new tasks with just a few demonstrations in the input context. However, the
model's ability to perform ICL is sensitive to the choice of the few-shot
demonstrations. Instead of using a fixed set of demonstrations, one recent
development is to retrieve demonstrations tailored to each input query. The
implementation of demonstration retrieval is relatively straightforward,
leveraging existing databases and retrieval systems. This not only improves the
efficiency and scalability of the learning process but also has been shown to
reduce biases inherent in manual example selection. In light of the encouraging
results and growing research in ICL with retrieved demonstrations, we conduct
an extensive review of studies in this area. In this survey, we discuss and
compare different design choices for retrieval models, retrieval training
procedures, and inference algorithms.

语言模型特别是预训练大型语言模型，在上下文少例学习方面表现出非凡的能力，能够在输入上下文中仅通过几个示例适应新任务。最近的一个发展是使用对每个输入查询量身定制的示例进行检索，这不仅提高了学习过程的效率和可伸缩性，还能减少手动示例选择中固有的偏差。基于鼓舞人心的结果和对检索式少例学习领域日益增长的研究，我们对该领域的研究进行了广泛的概述回顾，在此调查中，我们讨论并比较了检索模型、检索训练程序和推理算法的不同设计选择。

基于检索演示的语言模型的上下文学习：一项调查

In-context Learning with Retrieved Demonstrations for Language Models: A  Survey

Large language models show impressive results on few-shot NLP tasks. However,
these models are memory and computation-intensive. Meta-training allows one to
leverage smaller models for few-shot generalization in a domain-general and
task-agnostic manner; however, these methods alone results in models that may
not have sufficient parameterization or knowledge to adapt quickly to a large
variety of tasks. To overcome this issue, we propose meta-training with
demonstration retrieval, where we use a dense passage retriever to retrieve
semantically similar labeled demonstrations to each example for more varied
supervision. By separating external knowledge from model parameters, we can use
meta-training to train parameter-efficient models that generalize well on a
larger variety of tasks. We construct a meta-training set from UnifiedQA and
CrossFit, and propose a demonstration bank based on UnifiedQA tasks. To our
knowledge, our work is the first to combine retrieval with meta-training, to
use DPR models to retrieve demonstrations, and to leverage demonstrations from
many tasks simultaneously, rather than randomly sampling demonstrations from
the training set of the target task. Our approach outperforms a variety of
targeted parameter-efficient and retrieval-augmented few-shot methods on QA,
NLI, and text classification tasks (including SQuAD, QNLI, and TREC). Our
approach can be meta-trained and fine-tuned quickly on a single GPU.

本研究提出了一种基于演示检索的元训练方法，使用密集的段落检索器检索出与示例相似的标注演示，用于元训练，以训练参数高效的模型，从而在更广泛的 NLP 任务中实现更好的泛化性能。

利用演示检索进行元训练，以实现高效的少样本学习

Meta-training with Demonstration Retrieval for Efficient Few-shot  Learning

In-context learning is a new learning paradigm where a language model
conditions on a few input-output pairs (demonstrations) and a test input, and
directly outputs the prediction. It has been shown highly dependent on the
provided demonstrations and thus promotes the research of demonstration
retrieval: given a test input, relevant examples are retrieved from the
training set to serve as informative demonstrations for in-context learning.
While previous works focus on training task-specific retrievers for several
tasks separately, these methods are often hard to transfer and scale on various
tasks, and separately trained retrievers incur a lot of parameter storage and
deployment cost. In this paper, we propose Unified Demonstration Retriever
(\textbf{UDR}), a single model to retrieve demonstrations for a wide range of
tasks. To train UDR, we cast various tasks' training signals into a unified
list-wise ranking formulation by language model's feedback. Then we propose a
multi-task list-wise ranking training framework, with an iterative mining
strategy to find high-quality candidates, which can help UDR fully incorporate
various tasks' signals. Experiments on 30+ tasks across 13 task families and
multiple data domains show that UDR significantly outperforms baselines.
Further analyses show the effectiveness of each proposed component and UDR's
strong ability in various scenarios including different LMs (1.3B - 175B),
unseen datasets, varying demonstration quantities, etc.

在这篇论文中，我们提出了一种统一的演示检索器（Unified Demonstration Retriever，UDR），通过语言模型的反馈，将各种任务的训练信号转化为一个统一的列表排序公式，并采用迭代挖掘策略来找到高质量的候选人，UDR 在 30 多项任务和 13 个任务系列以及多个数据领域上的实验表明，它显著优于基准。