The in-context learning capabilities of LLMs like GPT-3 allow annotators to customize an LLM to their specific tasks with a small number of examples. However, users tend to include only the most obvious patterns when crafting examples, resulting in underspecified in-context functions that fall short on unseen cases. Further, it is hard to know when "enough" examples have been included even for known patterns. In this work, we present ScatterShot, an interactive system for building high-quality demonstration sets for in-context learning. ScatterShot iteratively slices unlabeled data into task-specific patterns, samples informative inputs from underexplored or not-yet-saturated slices in an active learning manner, and helps users label more efficiently with the help of an LLM and the current example set. In simulation studies on two text perturbation scenarios, ScatterShot sampling improves the resulting few-shot functions by 4-5 percentage points over random sampling, with less variance as more examples are added. In a user study, ScatterShot greatly helps users in covering different patterns in the input space and labeling in-context examples more efficiently, resulting in better in-context learning and less user effort.

本研究提出了ScatterShot，这是一个互动系统，可用于构建高质量的示范集以进行上下文学习，该系统通过在任务特定的模式中切片未标记数据，并以主动学习方式从未经探索或未饱和的切片中采样信息输入，同时利用LLM和当前的示例集帮助用户更有效地进行标记。在两个文本扰动场景的模拟研究中，ScatterShot采样将导致结果为4-5个百分点的提高，在添加更多示例的情况下方差更小。在用户研究中，ScatterShot极大地帮助用户覆盖输入空间中的不同模式，更有效地标记上下文示例，从而实现更好的上下文学习和更少的用户工作量。

ScatterShot: 文本转换互动示例筛选