Both humans and transformer language models are able to learn language
without explicit structural supervision. What inductive learning biases make
this learning possible? In this study, we examine the effect of different
inductive learning biases by predisposing language models with structural
biases through pretraining on artificial structured data, and then evaluating
by fine-tuning on English. Our experimental setup gives us the ability to
actively control the inductive bias of language models. With our experiments,
we investigate the comparative success of three types of inductive bias: 1) an
inductive bias for recursive, hierarchical processing 2) an inductive bias for
unrestricted token-token dependencies that can't be modeled by context-free
grammars, and 3) an inductive bias for a Zipfian power-law vocabulary
distribution. We show that complex token-token interactions form the best
inductive biases, and that this is strongest in the non-context-free case. We
also show that a Zipfian vocabulary distribution forms a good inductive bias
independently from grammatical structure. Our study leverages the capabilities
of transformer models to run controlled language learning experiments that are
not possible to run in humans, and surfaces hypotheses about the structures
that facilitate language learning in both humans and machines.

本研究通过对语言模型进行先验的结构性偏置，探索了不同感性学习偏差的影响，并研究了三种感性偏差的相对成功：1）递归，分层处理的感性偏差；2）无法通过上下文自由文法建模的无限制令牌 - 令牌依赖的感性偏差；和 3）一个 Zipfian 幂律词汇分布的感性偏差。我们发现，复杂的令牌 - 令牌交互形成了最好的感性偏向，并且在非上下文自由情况下最强。同时，我们还表明，独立于语法结构，Zipfian 词汇分布形成一个良好的感性偏向。