We know very little about how neural language models (LM) use prior
linguistic context. In this paper, we investigate the role of context in an
LSTM LM, through ablation studies. Specifically, we analyze the increase in
perplexity when prior context words are shuffled, replaced, or dropped. On two
standard datasets, Penn Treebank and WikiText-2, we find that the model is
capable of using about 200 tokens of context on average, but sharply
distinguishes nearby context (recent 50 tokens) from the distant history. The
model is highly sensitive to the order of words within the most recent
sentence, but ignores word order in the long-range context (beyond 50 tokens),
suggesting the distant past is modeled only as a rough semantic field or topic.
We further find that the neural caching model (Grave et al., 2017b) especially
helps the LSTM to copy words from within this distant context. Overall, our
analysis not only provides a better understanding of how neural LMs use their
context, but also sheds light on recent success from cache-based models.

本研究通过切片研究分析，探讨了神经语言模型如何利用其先前的语境。结果显示，模型能有效地利用大约 200 个标记的语境，但对远古语境的词序并不敏感，而是将其建模为一个粗糙的语义领域或主题，同时这项研究还为基于缓存的模型的最近的成功提供了启示。