Both humans and transformer language models are able to learn language
without explicit structural supervision. What inductive learning biases make
this learning possible? In this study, we examine the effect of different
inductive learning biases by predisposing language models with structural
biases through pretraining on artificial structured data, and then evaluating
by fine-tuning on English. Our experimental setup gives us the ability to
actively control the inductive bias of language models. With our experiments,
we investigate the comparative success of three types of inductive bias: 1) an
inductive bias for recursive, hierarchical processing 2) an inductive bias for
unrestricted token-token dependencies that can't be modeled by context-free
grammars, and 3) an inductive bias for a Zipfian power-law vocabulary
distribution. We show that complex token-token interactions form the best
inductive biases, and that this is strongest in the non-context-free case. We
also show that a Zipfian vocabulary distribution forms a good inductive bias
independently from grammatical structure. Our study leverages the capabilities
of transformer models to run controlled language learning experiments that are
not possible to run in humans, and surfaces hypotheses about the structures
that facilitate language learning in both humans and machines.

本研究通过对语言模型进行先验的结构性偏置，探索了不同感性学习偏差的影响，并研究了三种感性偏差的相对成功：1）递归，分层处理的感性偏差；2）无法通过上下文自由文法建模的无限制令牌 - 令牌依赖的感性偏差；和 3）一个 Zipfian 幂律词汇分布的感性偏差。我们发现，复杂的令牌 - 令牌交互形成了最好的感性偏向，并且在非上下文自由情况下最强。同时，我们还表明，独立于语法结构，Zipfian 词汇分布形成一个良好的感性偏向。

只预训结构：利用迁移学习理解语言归纳偏差

Pretrain on just structure: Understanding linguistic inductive biases  using transfer learning

The ability to learn and predict simple functions is a key aspect of human
intelligence. Recent works have started to explore this ability using
transformer architectures, however it remains unclear whether this is
sufficient to recapitulate the extrapolation abilities of people in this
domain. Here, we propose to address this gap by augmenting the transformer
architecture with two simple inductive learning biases, that are directly
adapted from recent models of abstract reasoning in cognitive science. The
results we report demonstrate that these biases are helpful in the context of
large neural network models, as well as shed light on the types of inductive
learning biases that may contribute to human abilities in extrapolation.

本研究通过向 Transformer 架构添加两种归纳学习偏差来探究个体的外推能力，结果表明，这些偏差对大型神经网络模型有帮助，也揭示了人类的归纳性学习偏差的类型。