We propose an alternate approach to quantifying how well language models
learn natural language: we ask how well they match the statistical tendencies
of natural language. To answer this question, we analyze whether text generated
from language models exhibits the statistical tendencies present in the
human-generated text on which they were trained. We provide a framework--paired
with significance tests--for evaluating the fit of language models to these
trends. We find that neural language models appear to learn only a subset of
the tendencies considered, but align much more closely with empirical trends
than proposed theoretical distributions (when present). Further, the fit to
different distributions is highly-dependent on both model architecture and
generation strategy. As concrete examples, text generated under the nucleus
sampling scheme adheres more closely to the type--token relationship of natural
language than text produced using standard ancestral sampling; text from LSTMs
reflects the natural language distributions over length, stopwords, and symbols
surprisingly well.

我们提出了一个替代方法来量化语言模型学习自然语言的程度：我们询问它们多大程度上与自然语言的统计倾向相匹配。通过分析语言模型生成的文本是否呈现出它们所训练的人类生成文本中存在的统计倾向，提供了一个与显著性测试配对的框架来评估语言模型的拟合程度。我们发现神经语言模型似乎只学会了一部分被考虑的倾向，但与经验性趋势相比，更接近所提出的理论分布（当存在时）。此外，对不同分布的拟合程度高度依赖于模型架构和生成策略。作为具体例子，使用 Nucleus sampling 方式生成的文本比使用标准祖先抽样生成的文本更紧密地遵循自然语言的类型 - 标记关系；LSTMs 生成的文本也非常好地反映了自然语言的长度、停用词和符号分布。