Large Language Models (LLM) hold immense promise for real-world applications,
but their generic knowledge often falls short of domain-specific needs.
Fine-tuning, a common approach, can suffer from catastrophic forgetting and
hinder generalizability. In-Context Learning (ICL) offers an alternative, which
can leverage Retrieval-Augmented Generation (RAG) to provide LLMs with relevant
demonstrations for few-shot learning tasks. This paper explores the desired
qualities of a demonstration retrieval system for ICL. We argue that ICL
retrieval in this context resembles item-cold-start recommender systems,
prioritizing discovery and maximizing information gain over strict relevance.
We propose a novel evaluation method that measures the LLM's subsequent
performance on NLP tasks, eliminating the need for subjective diversity scores.
Our findings demonstrate the critical role of diversity and quality bias in
retrieved demonstrations for effective ICL, and highlight the potential of
recommender system techniques in this domain.

大语言模型和上下文学习在创建演示检索系统方面具有关键作用，该系统利用检索增强生成为少样本学习任务提供相关演示，通过提出一种新的评估方法，探讨了多样性和质量偏差对有效上下文学习的关键作用，并突出了推荐系统技术在该领域的潜力。