We investigate how neural language models acquire individual words during
training, extracting learning curves and ages of acquisition for over 600 words
on the MacArthur-Bates Communicative Development Inventory (Fenson et al.,
2007). Drawing on studies of word acquisition in children, we evaluate multiple
predictors for words' ages of acquisition in LSTMs, BERT, and GPT-2. We find
that the effects of concreteness, word length, and lexical class are pointedly
different in children and language models, reinforcing the importance of
interaction and sensorimotor experience in child language acquisition. Language
models rely far more on word frequency than children, but like children, they
exhibit slower learning of words in longer utterances. Interestingly, models
follow consistent patterns during training for both unidirectional and
bidirectional models, and for both LSTM and Transformer architectures. Models
predict based on unigram token frequencies early in training, before
transitioning loosely to bigram probabilities, eventually converging on more
nuanced predictions. These results shed light on the role of distributional
learning mechanisms in children, while also providing insights for more
human-like language acquisition in language models.

研究神经语言模型在训练过程中如何获取单词，并提取 MacArthur-Bates 交际发展清单上 600 多个单词的学习曲线和获取年龄。与儿童的单词获取研究结合，评估 LSTMs、BERT 和 GPT-2 模型的多种预测因素，发现模型对单词频率的依赖程度远高于儿童，但与儿童相似的是，它们在更长的话语中学习单词较慢。同时，模型在训练过程中遵循一致的模式，这些结果为人类语言模型的更类人化语言习得提供了启示。