In-context learning is a new learning paradigm where a language model
observes a few examples and then straightly outputs the test input's
prediction. Previous works have shown that in-context learning is sensitive to
the provided examples and randomly sampled examples show significantly unstable
performance. In this paper, we propose to find ``supporting examples'' for
in-context learning: Given the training dataset, we need to select one
permutation of a few examples, which are informative for the task's in-context
learning and lead to superior performance. Although in traditional
gradient-based learning, e.g., fine-tuning, there are numerous methods to find
a ``coreset'' from the entire dataset, they are sub-optimal and not suitable
for this problem since in-context learning occurs in the language model's
inference without gradients or parameter updates. Additionally, the strong
dependence among in-context examples makes this problem an NP-hard
combinatorial optimization problem and enumerating all possible permutations is
infeasible. Hence we propose a two-stage method to tackle this challenge. First
we propose a novel metric to select informative examples based on the language
model's feedback, with a progressive filtering strategy. And then we propose a
diversity-guided beam search method to refine and evaluate the selected
examples, iteratively. The experimental results show our method significantly
outperforms a wide range of baselines, and further analyses show the
effectiveness of our method and shed light on the properties of supporting
examples and in-context learning.

本文提出了一种基于语言模型的支持性示例挑选方案，通过两阶段的方法解决了 NP-hard 组合优化问题并使用多样性引导的波束搜索方法优化选定示例的性能，进一步实验证明了我们方法的有效性和 “支持性示例” 和 “上下文学习” 的相关特征。