In-context learning (ICL) enables large language models (LLMs) to perform new
tasks by prompting them with a sequence of training examples. However, ICL is
very sensitive to the choice of training examples: randomly sampling examples
from a training set leads to high variance in performance. In this paper, we
show that curating a carefully chosen subset of training data greatly
stabilizes ICL performance. We propose two methods to choose training subsets,
both of which score training examples individually and then select the
highest-scoring ones. CondAcc scores a training example by its average ICL
accuracy when combined with random training examples, while Datamodels learns a
linear proxy model that estimates how the presence of each training example
influences LLM accuracy. On average, CondAcc and Datamodels outperform sampling
from the entire training set by 7.7% and 6.3%, respectively, across 5 tasks and
two LLMs. Our analysis shows that stable subset examples are no more diverse
than average, and are not outliers in terms of sequence length and perplexity.

本文提出两种方法优化 ICL 在训练过程中因训练样本选择不当而导致的性能不稳定问题：CondAcc 和 Datamodels。这两种方法均通过对训练样本进行评分，然后选择得分最高的样本作为训练子集。在五个任务和两个 LLMs 上进行的实验中，CondAcc 和 Datamodels 的性能分别比从整个训练集中随机抽样高出 7.7％和 6.3％。分析表明，选出的稳定子集样本与平均样本的差异不大，并不是序列长度和 perplexity 的异常值。