We explore the mechanism of in-context learning and propose a hypothesis
using locate-and-project method. In shallow layers, the features of
demonstrations are merged into their corresponding labels, and the features of
the input text are aggregated into the last token. In deep layers, in-context
heads make great contributions. In each in-context head, the value-output
matrix extracts the labels' features. Query and key matrices compute the
attention weights between the input text and each demonstration. The larger the
attention weight is, the more label information is transferred into the last
token for predicting the next word. Query and key matrices can be regarded as
two towers for learning the similarity metric between the input text and each
demonstration. Based on this hypothesis, we explain why imbalanced labels and
demonstration order affect predictions. We conduct experiments on GPT2 large,
Llama 7B, 13B and 30B. The results can support our analysis. Overall, our study
provides a new method and a reasonable hypothesis for understanding the
mechanism of in-context learning. Our code will be released on github.

我们通过定位和投射方法探索了上下文学习的机制，并提出了一种假设。在浅层中，合并演示的特征到对应的标签中，将输入文本的特征聚合到最后一个标记中。在深层中，上下文头起到了重要作用。在每个上下文头中，值输出矩阵提取标签的特征。查询和键矩阵计算输入文本与每个演示之间的注意权重。注意权重越大，越多的标签信息传递到最后一个标记以预测下一个单词。查询和键矩阵可以被视为学习输入文本与每个演示之间相似度的两个塔。基于这个假设，我们解释了不平衡的标签和演示顺序对预测的影响。我们在 GPT2 large，Llama 7B，13B 和 30B 上进行了实验。结果支持我们的分析。总体而言，我们的研究为理解上下文学习的机制提供了一种新的方法和合理的假设。我们的代码将在 GitHub 上发布。