In-Context Learning (ICL) is a critical capability of Large Language Models
(LLMs) as it empowers them to comprehend and reason across interconnected
inputs. Evaluating the ICL ability of LLMs can enhance their utilization and
deepen our understanding of how this ability is acquired at the training stage.
However, existing evaluation frameworks primarily focus on language abilities
and knowledge, often overlooking the assessment of ICL ability. In this work,
we introduce the ICLEval benchmark to evaluate the ICL abilities of LLMs, which
encompasses two key sub-abilities: exact copying and rule learning. Through the
ICLEval benchmark, we demonstrate that ICL ability is universally present in
different LLMs, and model size is not the sole determinant of ICL efficacy.
Surprisingly, we observe that ICL abilities, particularly copying, develop
early in the pretraining process and stabilize afterward. Our source codes and
benchmark are released at this https URL

通过 ICLEval 基准测试，我们展示了不同大语言模型中 ICL 能力普遍存在，并且模型大小不是 ICL 功效的唯一决定因素。