Large Language Models (LLMs) have demonstrated impressive in-context learning
(ICL) capabilities from few-shot demonstration exemplars. While recent
learning-based demonstration selection methods have proven beneficial to ICL by
choosing more useful exemplars, their underlying mechanisms are opaque,
hindering efforts to address limitations such as high training costs and poor
generalization across tasks. These methods generally assume the selection
process captures similarities between the exemplar and the target instance,
however, it remains unknown what kinds of similarities are captured and vital
to performing ICL. To dive into this question, we analyze the working
mechanisms of the learning-based demonstration selection methods and
empirically identify two important factors related to similarity measurement:
1) The ability to integrate different levels of task-agnostic text similarities
between the input of exemplars and test cases enhances generalization power
across different tasks. 2) Incorporating task-specific labels when measuring
the similarities significantly improves the performance on each specific task.
We validate these two findings through extensive quantitative and qualitative
analyses across ten datasets and various LLMs. Based on our findings, we
introduce two effective yet simplified exemplar selection methods catering to
task-agnostic and task-specific demands, eliminating the costly LLM inference
overhead.

通过分析学习基于示例选择方法的工作机制，我们鉴定出与相似度测量相关的两个重要因素：1）在示例和测试用例输入之间整合不同级别的与任务无关的文本相似度能够增强在不同任务中的泛化能力；2）在测量相似度时结合任务特定标签显著提高每个特定任务的性能。基于这些发现，我们提出了两种适应任务无关和任务特定需求的有效简化示例选择方法，消除了昂贵的大型语言模型推理开销。