The usual way to interpret language models (LMs) is to test their performance
on different benchmarks and subsequently infer their internal processes. In
this paper, we present an alternative approach, concentrating on the quality of
LM processing, with a focus on their language abilities. To this end, we
construct 'linguistic task spaces' -- representations of an LM's language
conceptualisation -- that shed light on the connections LMs draw between
language phenomena. Task spaces are based on the interactions of the learning
signals from different linguistic phenomena, which we assess via a method we
call 'similarity probing'. To disentangle the learning signals of linguistic
phenomena, we further introduce a method called 'fine-tuning via gradient
differentials' (FTGD). We apply our methods to language models of three
different scales and find that larger models generalise better to overarching
general concepts for linguistic tasks, making better use of their shared
structure. Further, the distributedness of linguistic processing increases with
pre-training through increased parameter sharing between related linguistic
tasks. The overall generalisation patterns are mostly stable throughout
training and not marked by incisive stages, potentially explaining the lack of
successful curriculum strategies for LMs.

通过构建语言任务空间，借助相似性探测与梯度差分的微调方法，研究发现大型语言模型更好地泛化到语言任务的总体概念，利用其共享结构。此外，预训练通过加强相关语言任务之间的参数共享来增加语言处理的分布性。整体泛化模式在训练过程中基本稳定且没有明显分界点，这可能解释了语言模型缺乏成功的课程策略的原因。