The emergence of in-context learning (ICL) is potentially attributed to two
major abilities: task recognition (TR) for recognizing the task from
demonstrations and utilizing pre-trained priors, and task learning (TL) for
learning from demonstrations. However, relationships between the two abilities
and how such relationships affect the emergence of ICL is unclear. In this
paper, we take the first step by examining the pre-training dynamics of the
emergence of ICL. With carefully designed metrics, we find that these two
abilities are, in fact, competitive during pre-training. Moreover, we observe a
strong negative correlation between the competition and ICL performance.
Further analysis of common pre-training factors (i.e., model size, dataset
size, and data curriculum) demonstrates possible ways to manage the
competition. Based on these insights, we propose a simple yet effective method
to better integrate these two abilities for ICL at inference time. Through
adaptive ensemble learning, the performance of ICL can be significantly
boosted, enabling two small models to outperform a larger one with more than
twice the parameters. The code is available at
this https URL

利用预先训练的先验知识，通过任务识别和任务学习两种能力的竞争来促进上下文学习的出现，并提出了一种简单而有效的方法，在推理时间内更好地整合这两种能力。通过自适应集成学习，可以显著提高上下文学习的性能，使两个小模型的性能超过具有两倍参数的大模型。

探究上下文学习的预训练动态：任务识别与任务学习

Investigating the Pre-Training Dynamics of In-Context Learning: Task  Recognition vs. Task Learning

Self-supervised large language models have demonstrated the ability to
perform Machine Translation (MT) via in-context learning, but little is known
about where the model performs the task with respect to prompt instructions and
demonstration examples. In this work, we attempt to characterize the region
where large language models transition from in-context learners to translation
models. Through a series of layer-wise context-masking experiments on
\textsc{GPTNeo2.7B}, \textsc{Bloom3B}, \textsc{Llama7b} and
\textsc{Llama7b-chat}, we demonstrate evidence of a "task recognition" point
where the translation task is encoded into the input representations and
attention to context is no longer necessary. We further observe correspondence
between the low performance when masking out entire layers, and the task
recognition layers. Taking advantage of this redundancy results in 45\%
computational savings when prompting with 5 examples, and task recognition
achieved at layer 14 / 32. Our layer-wise fine-tuning experiments indicate that
the most effective layers for MT fine-tuning are the layers critical to task
recognition.

通过层次上的上下文遮蔽实验，我们证明了大型语言模型中存在一个任务识别点，该点将任务编码到输入表示中，不再需要注意上下文，同时还观察到在层次遮蔽时的低性能与任务识别层之间的对应关系，利用这种冗余性可在提示 5 个示例时节省 45% 的计算量，任务识别在第 14/32 层达到，并且层次微调实验表明对于 MT 微调来说，最有效的层次是关键的任务识别层。

大型语言模型中的上下文翻译发生在哪里

Where does In-context Translation Happen in Large Language Models

Large language models (LLMs) exploit in-context learning (ICL) to solve tasks
with only a few demonstrations, but its mechanisms are not yet well-understood.
Some works suggest that LLMs only recall already learned concepts from
pre-training, while others hint that ICL performs implicit learning over
demonstrations. We characterize two ways through which ICL leverages
demonstrations. Task recognition (TR) captures the extent to which LLMs can
recognize a task through demonstrations -- even without ground-truth labels --
and apply their pre-trained priors, whereas task learning (TL) is the ability
to capture new input-label mappings unseen in pre-training. Using a wide range
of classification datasets and three LLM families (GPT-3, LLaMA and OPT), we
design controlled experiments to disentangle the roles of TR and TL in ICL. We
show that (1) models can achieve non-trivial performance with only TR, and TR
does not further improve with larger models or more demonstrations; (2) LLMs
acquire TL as the model scales, and TL's performance consistently improves with
more demonstrations in context. Our findings unravel two different forces
behind ICL and we advocate for discriminating them in future ICL research due
to their distinct nature.

本研究探讨了大型语言模型利用情境学习来解决只有少量演示的任务的机制，发现情境学习从两个方面发挥作用：任务识别和任务学习，具有不同的性质和特点。