Few shot in-context learning (ICL) typically assumes access to large
annotated training sets. However, in many real world scenarios, such as domain
adaptation, there is only a limited budget to annotate a small number of
samples, with the goal of maximizing downstream performance. We study various
methods for selecting samples to annotate within a predefined budget,
specifically focusing on the named entity recognition (NER) task, which has
real-world applications, is expensive to annotate, and is relatively less
studied in ICL setups. Across different models and datasets, we find that a
relatively small pool of annotated samples can achieve results comparable to
using the entire training set. Moreover, we discover that random selection of
samples for annotation yields surprisingly good performance. Finally, we
observe that a diverse annotation pool is correlated with improved performance.
We hope that future work adopts our realistic paradigm which takes annotation
budget into account.

在有限的预算内，本研究通过研究不同方法选择样本进行注释，发现少样本内上下文学习（ICL）可以在命名实体识别（NER）任务中实现与使用完整训练集相当的结果，而随机选择样本进行注释可以获得出乎意料的良好性能，并且多样化的注释池与性能的提升相关，希望未来的工作将我们的现实范式纳入考量。