When generating natural language from neural probabilistic models, high
probability does not always coincide with high quality: It has often been
observed that mode-seeking decoding methods, i.e., those that produce
high-probability text under the model, lead to unnatural language. On the other
hand, the lower-probability text generated by stochastic methods is perceived
as more human-like. In this note, we offer an explanation for this phenomenon
by analyzing language generation through an information-theoretic lens.
Specifically, we posit that human-like language should contain an amount of
information (quantified as negative log-probability) that is close to the
entropy of the distribution over natural strings. Further, we posit that
language with substantially more (or less) information is undesirable. We
provide preliminary empirical evidence in favor of this hypothesis; quality
ratings of both human and machine-generated text -- covering multiple tasks and
common decoding strategies -- suggest high-quality text has an information
content significantly closer to the entropy than we would expect by chance.

论文分析了自然语言生成过程，并通过信息理论解释了高概率和高质量并不总是一致的现象，得出语言生成应包含接近自然字符串分布熵的负对数概率数目，初步实证结果表明高质量的文本具有信息量与自然字符串分布熵更接近的特点。