We describe our team's contribution to the STRICT-SMALL track of the BabyLM
Challenge. The challenge requires training a language model from scratch using
only a relatively small training dataset of ten million words. We experiment
with three variants of cognitively-motivated curriculum learning and analyze
their effect on the performance of the model on linguistic evaluation tasks. In
the vocabulary curriculum, we analyze methods for constraining the vocabulary
in the early stages of training to simulate cognitively more plausible learning
curves. In the data curriculum experiments, we vary the order of the training
instances based on i) infant-inspired expectations and ii) the learning
behavior of the model. In the objective curriculum, we explore different
variations of combining the conventional masked language modeling task with a
more coarse-grained word class prediction task to reinforce linguistic
generalization capabilities. Our results did not yield consistent improvements
over our own non-curriculum learning baseline across a range of linguistic
benchmarks; however, we do find marginal gains on select tasks. Our analysis
highlights key takeaways for specific combinations of tasks and settings which
benefit from our proposed curricula. We moreover determine that careful
selection of model architecture, and training hyper-parameters yield
substantial improvements over the default baselines provided by the BabyLM
challenge.

我们通过三种认知引导的课程学习变体对模型在语言评估任务上的性能进行了分析，并发现在早期训练阶段对词汇、训练实例的顺序和目标任务进行合理选择可以达到一定的改进效果。