We present Lil-Bevo, our submission to the BabyLM Challenge. We pretrained
our masked language models with three ingredients: an initial pretraining with
music data, training on shorter sequences before training on longer ones, and
masking specific tokens to target some of the BLiMP subtasks. Overall, our
baseline models performed above chance, but far below the performance levels of
larger LLMs trained on more data. We found that training on short sequences
performed better than training on longer sequences.Pretraining on music may
help performance marginally, but, if so, the effect seems small. Our targeted
Masked Language Modeling augmentation did not seem to improve model performance
in general, but did seem to help on some of the specific BLiMP tasks that we
were targeting (e.g., Negative Polarity Items). Training performant LLMs on
small amounts of data is a difficult but potentially informative task. While
some of our techniques showed some promise, more work is needed to explore
whether they can improve performance more than the modest gains here. Our code
is available at this https URL and out models at
this https URL

我们提出 Lil-Bevo，作为 BabyLM Challenge 的参赛作品。我们使用三种方法对我们的掩码语言模型进行了预训练：使用音乐数据进行初始预训练，先对较短的序列进行训练再逐步增加序列长度，以及对特定令牌进行屏蔽以针对 BLiMP 的一些子任务。总体上，我们的基准模型的表现超过了随机猜测，但远低于训练于更多数据上的较大规模语言模型的性能水平。我们发现训练较短的序列比训练较长的序列效果更好。预训练音乐可能在一定程度上提高性能，但如果确实有影响，则影响似乎很小。我们针对性的掩码语言建模增强在一般情况下似乎并没有提高模型性能，但在一些特定的 BLiMP 任务上确实有帮助（如负极性项）。在少量数据上训练性能良好的语言模型是一项困难但有潜力的任务。虽然我们的一些技术显示出一些希望，但仍需要进一步研究以探索它们是否能带来比目前更为显著的性能提升。我们的代码可在此链接中获取，模型可在此链接中获取。