In this paper, we describe our submission to the BabyLM Challenge 2023 shared
task on data-efficient language model (LM) pretraining (Warstadt et al., 2023).
We train transformer-based masked language models that incorporate unsupervised
predictions about hierarchical sentence structure into the model architecture.
Concretely, we use the Structformer architecture (Shen et al., 2021) and
variants thereof. StructFormer models have been shown to perform well on
unsupervised syntactic induction based on limited pretraining data, and to
yield performance improvements over a vanilla transformer architecture (Shen et
al., 2021). Evaluation of our models on 39 tasks provided by the BabyLM
challenge shows promising improvements of models that integrate a hierarchical
bias into the architecture at some particular tasks, even though they fail to
consistently outperform the RoBERTa baseline model provided by the shared task
organizers on all tasks.

在这篇文章中，我们描述了我们在 2023 年 BabyLM 挑战赛上对数据高效的语言模型（LM）预训练的提交。我们使用结构型语言模型架构以及其变种，将有关层次化句子结构的无监督预测纳入模型架构中，并在 39 个 BabyLM 挑战赛提供的任务上进行了评估，结果显示在某些特定任务上，将层次化偏差整合到架构中的模型表现出有希望的改进，尽管它们未能在所有任务上始终超过挑战赛组织者提供的 RoBERTa 基准模型。

通过隐式结构建立提高认知启发型数据有效语言模型的性能

Increasing The Performance of Cognitively Inspired Data-Efficient  Language Models via Implicit Structure Building

Syntactic rules in natural language typically need to make reference to
hierarchical sentence structure. However, the simple examples that language
learners receive are often equally compatible with linear rules. Children
consistently ignore these linear explanations and settle instead on the correct
hierarchical one. This fact has motivated the proposal that the learner's
hypothesis space is constrained to include only hierarchical rules. We examine
this proposal using recurrent neural networks (RNNs), which are not constrained
in such a way. We simulate the acquisition of question formation, a
hierarchical transformation, in a fragment of English. We find that some RNN
architectures tend to learn the hierarchical rule, suggesting that hierarchical
cues within the language, combined with the implicit architectural biases
inherent in certain RNNs, may be sufficient to induce hierarchical
generalizations. The likelihood of acquiring the hierarchical generalization
increased when the language included an additional cue to hierarchy in the form
of subject-verb agreement, underscoring the role of cues to hierarchy in the
learner's input.

本研究使用循环神经网络探讨了语言学习者偏好使用分层规则的原因，发现循环神经网络架构的隐含偏见和语言内在层次结构线索结合可以诱导分层概括规则的习得。