The performance of Large Language Models (LLMs) on downstream tasks often
improves significantly when including examples of the input-label relationship
in the context. However, there is currently no consensus about how this
in-context learning (ICL) ability of LLMs works: for example, while Xie et al.
(2021) liken ICL to a general-purpose learning algorithm, Min et al. (2022b)
argue ICL does not even learn label relationships from in-context examples. In
this paper, we study (1) how labels of in-context examples affect predictions,
(2) how label relationships learned during pre-training interact with
input-label examples provided in-context, and (3) how ICL aggregates label
information across in-context examples. Our findings suggests LLMs usually
incorporate information from in-context labels, but that pre-training and
in-context label relationships are treated differently, and that the model does
not consider all in-context information equally. Our results give insights into
understanding and aligning LLM behavior.

通过研究源自于上下文的学习能力、预训练和标签关系对大语言模型性能的影响，我们的研究发现 LLMs 通常会整合上下文标签信息，但预训练和上下文标签关系被区别对待。同时，模型并不会同等考虑所有上下文信息，这些发现有助于理解和调整 LLM 的行为。