Large language models (LLMs) have shown a remarkable ability to learn and perform complex tasks through in-context learning (ICL). However, a comprehensive understanding of its internal mechanisms is still lacking. This paper explores the role of induction heads in a few-shot ICL setting. We analyse two state-of-the-art models, Llama-3-8B and InternLM2-20B on abstract pattern recognition and NLP tasks. Our results show that even a minimal ablation of induction heads leads to ICL performance decreases of up to ~32% for abstract pattern recognition tasks, bringing the performance close to random. For NLP tasks, this ablation substantially decreases the model's ability to benefit from examples, bringing few-shot ICL performance close to that of zero-shot prompts. We further use attention knockout to disable specific induction patterns, and present fine-grained evidence for the role that the induction mechanism plays in ICL.

大语言模型通过上下文学习展现了学习和执行复杂任务的卓越能力，本文研究在少样本学习和上下文学习环境中的归纳头的作用，并在抽象模式识别和自然语言处理任务上分析了两种最先进的模型，Llama-3-8B和InternLM2-20B。研究结果表明，即使对归纳头进行最小的抽取也会导致抽象模式识别任务中ICL性能下降约32％，使性能接近随机水平。对于自然语言处理任务，该抽取显著降低了模型利于示例学习的能力，使得少样本学习在上下文学习中的表现接近于零样本提示学习。我们还使用注意力削减方法来禁用特定的归纳模式，并提供了对归纳机制在上下文学习中所起作用的细粒度证据。

上下文学习中的感应头作为模式匹配的基础机制