We explore the critical data size in language models, a threshold that marks
a fundamental shift from quick memorization to slow generalization. We
formalize the phase transition under the grokking configuration into the Data
Efficiency Hypothesis and identify data insufficiency, sufficiency, and surplus
regimes in language models training dynamics. We develop a grokking
configuration to reproduce grokking on simplistic language models stably by
rescaling initialization and weight decay. We show that generalization occurs
only when language models reach a critical size. We analyze grokking across
sample-wise and model-wise, verifying the proposed data efficiency hypothesis.
Our experiments reveal smoother phase transitions occurring at the critical
dataset size for language datasets. As the model size increases, this critical
point also becomes larger, indicating that larger models require more data. Our
results deepen the understanding of language model training, offering a novel
perspective on the role of data in the learning mechanism of language models.

通过研究数据规模，我们发现了语言模型从快速记忆到慢速泛化的临界点，提出了数据效率假设并确定了语言模型训练中的数据不足、充足和过剩阶段。我们开发了一种配置方法来稳定地在简化语言模型中达到这种理解。实验结果显示，只有当语言模型达到一定的关键尺寸时才会发生泛化，我们对样本和模型进行了分析，验证了数据效率假设。我们的研究深化了对语言模型训练的理解，并提供了一个新的视角来解释数据在语言模型学习机制中的作用。