To predict upcoming text, language models must in some cases retrieve in-context information verbatim. In this report, we investigated how the ability of language models to retrieve arbitrary in-context nouns developed during training (across time) and as language models trained on the same dataset increase in size (across scale). We then asked whether learning of in-context retrieval correlates with learning of more challenging zero-shot benchmarks. Furthermore, inspired by semantic effects in human short-term memory, we evaluated the retrieval with respect to a major semantic component of target nouns, namely whether they denote a concrete or abstract entity, as rated by humans. We show that verbatim in-context retrieval developed in a sudden transition early in the training process, after about 1% of the training tokens. This was observed across model sizes (from 14M and up to 12B parameters), and the transition occurred slightly later for the two smallest models. We further found that the development of verbatim in-context retrieval is positively correlated with the learning of zero-shot benchmarks. Around the transition point, all models showed the advantage of retrieving concrete nouns as opposed to abstract nouns. In all but two smallest models, the advantage dissipated away toward the end of training.

本研究解决了语言模型在预测文本时逐字检索上下文信息的能力发展问题。通过对模型训练过程的分析，我们发现这种逐字检索能力在训练早期的突然转变中出现，并与零样本基准的学习呈正相关。该研究的主要发现是，具体名词的检索优势在训练初期显著，但最终会逐渐减弱，这为理解语言模型的学习机制提供了重要见解。

跨时间和规模的变压器逐字上下文检索