In-context learning with large language models (LLMs) excels at adapting to
various tasks rapidly. However, its success hinges on carefully selecting
demonstrations, which remains an obstacle in practice. Current approaches to
this problem either rely on hard-to-acquire external supervision or require
frequent interactions with LLMs, resulting in high costs. We propose a new
method called In-Context Reflection (ICR) to overcome these challenges. ICR
strategically selects demonstrations to reduce the discrepancy between the
LLM's outputs and the actual input-output mappings. Specifically, ICR starts
with a random set of initial demonstrations, then iteratively refines it. In
each step, it analyzes a pool of candidate examples and identifies the ones
most likely to challenge the LLM's current understanding, measured by a new
metric called misconfidence. These most confusing examples are then selected to
replace the less informative demonstrations in the current set. Our
comprehensive evaluation across five diverse datasets encompassing 13 subtasks
shows the efficacy of ICR. Compared to existing methods, ICR achieves an
average performance boost of 4%, while demonstrating remarkable cross-task
generalization capabilities.

本文介绍了一种名为 In-Context Reflection (ICR) 的新方法，通过策略性地选择示范来减少大语言模型 (LLMs) 的输出与实际输入输出映射之间的差异，从而克服了在实践中选择示范的困难。经过综合评估，ICR 在五个不同数据集和 13 个子任务中显示出了显著的效果和跨任务的普适能力。