Large language models (LMs) have rapidly become a mainstay in Natural
Language Processing. These models are known to acquire rich linguistic
knowledge from training on large amounts of text. In this paper, we investigate
if pre-training on text also confers these models with helpful `inductive
biases' for non-linguistic reasoning. On a set of 19 diverse non-linguistic
tasks involving quantitative computations, recognizing regular expressions and
reasoning over strings. We find that pretrained models significantly outperform
comparable non-pretrained neural models. This remains true also in experiments
with training non-pretrained models with fewer parameters to account for model
regularization effects. We further explore the effect of text domain on LMs by
pretraining models from text from different domains and provenances. Our
experiments surprisingly reveal that the positive effects of pre-training
persist even when pretraining on multi-lingual text or computer code, and even
for text generated from synthetic languages. Our findings suggest a hitherto
unexplored deep connection between pre-training and inductive learning
abilities of language models.

本文调查了大型语言模型是否通过文本预训练会赋予这些模型有助于非语言推理的归纳偏差。通过对 19 个多样化的非语言任务进行试验，包括量化计算、识别正则表达式和对字符串的推理，我们发现预先训练的模型明显优于可比较的非预先训练的神经模型，即使在带有更少参数的情况下进行训练以适应模型正则化效应。同时，我们进一步探讨了不同文本领域对用户体验的影响，并发现即使在预训练多语言文本或计算机代码并生成合成语言的情况下，也可以持续预训练的积极效应。这些发现暗示了预先培训与模型的归纳学习能力之间的深层联系。