Large Language Models (LLMs), trained on massive corpora with billions of
parameters, show unprecedented performance in various fields. Though surprised
by their excellent performances, researchers also noticed some special
behaviors of those LLMs. One of those behaviors is memorization, in which LLMs
can generate the same content used to train them. Though previous research has
discussed memorization, the memorization of LLMs still lacks explanation,
especially the cause of memorization and the dynamics of generating them. In
this research, we comprehensively discussed memorization from various
perspectives and extended the discussion scope to not only just the memorized
content but also less and unmemorized content. Through various studies, we
found that: (1) Through experiments, we revealed the relation of memorization
between model size, continuation size, and context size. Further, we showed how
unmemorized sentences transition to memorized sentences. (2) Through embedding
analysis, we showed the distribution and decoding dynamics across model size in
embedding space for sentences with different memorization scores. The n-gram
statistics analysis presents d (3) An analysis over n-gram and entropy decoding
dynamics discovered a boundary effect when the model starts to generate
memorized sentences or unmemorized sentences. (4)We trained a Transformer model
to predict the memorization of different models, showing that it is possible to
predict memorizations by context.

大语言模型（LLMs）在各个领域展示了前所未有的性能，但其特殊行为之一 —— 记忆化 —— 仍缺乏解释，本研究通过多个角度全面探讨记忆化现象及其动态，并通过实验证实了模型大小、连续大小和上下文大小之间的记忆化关系，以及不同记忆化得分下句子的嵌入分布和解码动态，揭示了当模型开始生成记忆化或非记忆化句子时的边界效应，最后通过训练 Transformer 模型预测不同模型的记忆化，证明了通过上下文预测记忆化的可行性。