Text documents are structured on multiple levels of detail: individual words
are related by syntax, but larger units of text are related by discourse
structure. Existing language models generally fail to account for discourse
structure, but it is crucial if we are to have language models that reward
coherence and generate coherent texts. We present and empirically evaluate a
set of multi-level recurrent neural network language models, called
Document-Context Language Models (DCLM), which incorporate contextual
information both within and beyond the sentence. In comparison with word-level
recurrent neural network language models, the DCLM models obtain slightly
better predictive likelihoods, and considerably better assessments of document
coherence.

文本文件通常按多个层次结构化：单个单词由语法相关联，但更大的文本单元由话语结构相关联。本研究提出并实证评估了一组多级递归神经网络语言模型，称为文档上下文语言模型（DCLM），它们在句子内外都融入上下文信息。与基于单词级的递归神经网络语言模型相比，DCLM 模型预测概率略有提高，文档连贯性评估显著提高。