The proliferation of Large Language Models (LLMs) highlights the critical
importance of conducting thorough evaluations to discern their comparative
advantages, limitations, and optimal use cases. Particularly important is
assessing their capacity to accurately retrieve information included in a given
prompt. A model's ability to do this significantly influences how effectively
it can utilize contextual details, thus impacting its practical efficacy and
dependability in real-world applications.
Our research analyzes the in-context recall performance of various LLMs using
the needle-in-a-haystack method. In this approach, a factoid (the "needle") is
embedded within a block of filler text (the "haystack"), which the model is
asked to retrieve. We assess the recall performance of each model across
various haystack lengths and with varying needle placements to identify
performance patterns. This study demonstrates that an LLM's recall capability
is not only contingent upon the prompt's content but also may be compromised by
biases in its training data. Conversely, adjustments to model architecture,
training strategy, or fine-tuning can improve performance. Our analysis
provides insight into LLM behavior, offering direction for the development of
more effective applications of LLMs.

通过使用 needle-in-a-haystack 方法分析各种大型语言模型的上下文召回性能，我们的研究表明模型的成功检索能力不仅取决于提示内容，而且还可能受到训练数据中的偏见的影响。相反，通过对模型架构、训练策略或微调进行调整可以提高性能，我们的分析揭示了关于大型语言模型行为的见解，为开发更有效的应用提供了指导。