Fact knowledge memorization is crucial for Large Language Models (LLM) to
generate factual and reliable responses. However, the behaviors of LLM fact
memorization remain under-explored. In this paper, we analyze the scaling laws
for LLM's fact knowledge and LLMs' behaviors of memorizing different types of
facts. We find that LLMs' fact knowledge capacity has a linear and negative
exponential law relationship with model size and training epochs, respectively.
Estimated by the built scaling law, memorizing the whole Wikidata's facts
requires training an LLM with 1000B non-embed parameters for 100 epochs,
suggesting that using LLMs to memorize all public facts is almost implausible
for a general pre-training setting. Meanwhile, we find that LLMs can generalize
on unseen fact knowledge and its scaling law is similar to general
pre-training. Additionally, we analyze the compatibility and preference of
LLMs' fact memorization. For compatibility, we find LLMs struggle with
memorizing redundant facts in a unified way. Only when correlated facts have
the same direction and structure, the LLM can compatibly memorize them. This
shows the inefficiency of LLM memorization for redundant facts. For preference,
the LLM pays more attention to memorizing more frequent and difficult facts,
and the subsequent facts can overwrite prior facts' memorization, which
significantly hinders low-frequency facts memorization. Our findings reveal the
capacity and characteristics of LLMs' fact knowledge learning, which provide
directions for LLMs' fact knowledge augmentation.

通过分析大型语言模型（LLM）事实知识和 LLM 记忆不同类型事实的行为的扩展规律，我们发现 LLMs 的事实知识容量与模型规模和训练周期呈线性和负指数关系，认为 LLMs 用于记忆全部公共事实在普通预训练设置下几乎不切实际。与此同时，我们发现 LLMs 能够推广未见事实知识，并且其扩展规律类似于普通预训练。我们还分析了 LLMs 事实记忆的兼容性和偏好，结果表明 LLMs 在以统一方式记忆冗余事实时遇到困难，并且仅当相关事实具有相同的方向和结构时，LLM 才能兼容地记忆它们。此外，LLM 更关注记忆更频繁和更困难的事实，并且后续的事实可以覆盖先前事实的记忆，这严重阻碍了低频事实的记忆。我们发现了 LLMs 事实知识学习的容量和特点，为 LLMs 事实知识增强提供了方向。