Modeling human language requires the ability to not only generate fluent text
but also encode factual knowledge. However, traditional language models are
only capable of remembering facts seen at training time, and often have
difficulty recalling them. To address this, we introduce the knowledge graph
language model (KGLM), a neural language model with mechanisms for selecting
and copying facts from a knowledge graph that are relevant to the context.
These mechanisms enable the model to render information it has never seen
before, as well as generate out-of-vocabulary tokens. We also introduce the
Linked WikiText-2 dataset, a corpus of annotated text aligned to the Wikidata
knowledge graph whose contents (roughly) match the popular WikiText-2
benchmark. In experiments, we demonstrate that the KGLM achieves significantly
better performance than a strong baseline language model. We additionally
compare different language model's ability to complete sentences requiring
factual knowledge, showing that the KGLM outperforms even very large language
models in generating facts.

本研究提出了基于知识图谱的语言模型 (KGLM)，它通过从知识图谱中选择和复制与上下文相关的事实来生成准确的文本，在此基础上，引入了对齐 Wikidata 知识图谱的已注释文本的 Linked WikiText-2 数据集，通过实验比较，证明了 KGLM 在生成需要熟知事实的语句方面的性能优于现有的强基准语言模型。