In-context learning (ICL), the ability of large language models to perform
novel tasks by conditioning on a prompt with a few task examples, requires
demonstrations that are informative about the test instance. The standard
approach of independently selecting the most similar examples selects redundant
demonstrations while overlooking important information. This work proposes a
framework for assessing the informativeness of demonstrations based on their
coverage of salient aspects (e.g., reasoning patterns) of the test input. Using
this framework, we show that contextual token embeddings effectively capture
these salient aspects, and their recall measured using BERTScore-Recall (BSR)
yields a reliable measure of informativeness. Further, we extend recall metrics
like BSR to propose their set versions to find maximally informative sets of
demonstrations. On 6 complex compositional generation tasks and 7 diverse LLMs,
we show that Set-BSR outperforms the standard similarity-based approach by up
to 16% on average and, despite being learning-free, often surpasses methods
that leverage task or LLM-specific training.

该研究提出了一种基于覆盖测试输入相关重要方面的信息量评估框架，使用 Set-BSR 来找出信息最丰富的示范集，应用于复杂构成生成任务和多元语言模型中，得出与任务或模型特异性培训相比效果最佳的结果。