Recent advances in language modeling consist in pretraining highly
parameterized neural networks on extremely large web-mined text corpora.
Training and inference with such models can be costly in practice, which
incentivizes the use of smaller counterparts. However, it has been observed
that smaller models can suffer from saturation, characterized as a drop in
performance at some advanced point in training followed by a plateau. In this
paper, we find that such saturation can be explained by a mismatch between the
hidden dimension of smaller models and the high rank of the target contextual
probability distribution. This mismatch affects the performance of the linear
prediction head used in such models through the well-known softmax bottleneck
phenomenon. We measure the effect of the softmax bottleneck in various settings
and find that models based on less than 1000 hidden dimensions tend to adopt
degenerate latent representations in late pretraining, which leads to reduced
evaluation performance.

在语言建模中，研究人员发现较小的模型在训练后期可能会出现饱和现象，其原因是较小模型的隐藏维度与目标上下文概率分布的高秩之间存在不匹配，由此导致线性预测头的性能下降。实验结果表明，隐藏维度小于 1000 的模型在预训练后期会采用退化的潜在表示，从而导致评估性能降低。