Pre-trained language models (LMs) are capable of in-context learning (ICL):
they can adapt to a task with only a few examples given in the prompt without
any parameter update. However, it is unclear where this capability comes from
as there is a stark distribution shift between pre-training text and ICL
prompts. In this work, we study what patterns of the pre-training data
contribute to ICL. We find that LMs' ICL ability depends on $\textit{parallel
structures}$ in the pre-training data -- pairs of phrases following similar
templates in the same context window. Specifically, we detect parallel
structures by checking whether training on one phrase improves prediction of
the other, and conduct ablation experiments to study their effect on ICL. We
show that removing parallel structures in the pre-training data reduces LMs'
ICL accuracy by 51% (vs 2% from random ablation). This drop persists even when
excluding common patterns such as n-gram repetitions and long-range dependency,
showing the diversity and generality of parallel structures. A closer look at
the detected parallel structures indicates that they cover diverse linguistic
tasks and span long distances in the data.

通过检测训练模型的一对短语是否关注拓展，我们发现预训练数据中的平行结构在很大程度上影响预训练语言模型在不同上下文学习 (ICL) 任务中的准确性。移除这些平行结构会使得模型的 ICL 准确性下降 51%，这表明平行结构在 ICL 中的重要性和多样性。