We propose a new asymptotic equipartition property for the perplexity of a
large piece of text generated by a language model and present theoretical
arguments for this property. Perplexity, defined as a inverse likelihood
function, is widely used as a performance metric for training language models.
Our main result states that the logarithmic perplexity of any large text
produced by a language model must asymptotically converge to the average
entropy of its token distributions. This means that language models are
constrained to only produce outputs from a ``typical set", which we show, is a
vanishingly small subset of all possible grammatically correct outputs. We
present preliminary experimental results from an open-source language model to
support our theoretical claims. This work has possible practical applications
for understanding and improving ``AI detection" tools and theoretical
implications for the uniqueness, predictability and creative potential of
generative models.

我们提出了一个新的渐进均分特性来解析语言模型生成的大段文本的困惑度，并对该特性提出理论证明。困惑度，作为一个逆似然函数，被广泛用作训练语言模型的性能度量。我们的主要结果表明，由语言模型生成的任何大段文本的对数困惑度必须渐进地收敛到其令牌分布的平均熵。这意味着语言模型只能产生来自 “典型集合” 的输出，我们证明该集合是所有可能的语法正确的输出中一个逐渐变小的子集。我们提供了一个开源语言模型的初步实验结果来支持我们的理论论断。这项工作可能在理解和改进 “AI 检测” 工具方面具有实际应用，对生成模型的独特性、可预测性和创造潜力具有理论意义。