Inspired by the notion that ``{\it to copy is easier than to memorize}``, in this work, we introduce GNN-LM, which extends the vanilla neural language model (LM) by allowing to reference similar contexts in the entire training corpus. We build a directed heterogeneous graph between an input context and its semantically related neighbors selected from the training corpus, where nodes are tokens in the input context and retrieved neighbor contexts, and edges represent connections between nodes. Graph neural networks (GNNs) are constructed upon the graph to aggregate information from similar contexts to decode the token. This learning paradigm provides direct access to the reference contexts and helps improve a model's generalization ability. We conduct comprehensive experiments to validate the effectiveness of the GNN-LM: GNN-LM achieves a new state-of-the-art perplexity of 14.8 on WikiText-103 (a 4.5 point improvement over its counterpart of the vanilla LM model) and shows substantial improvement on One Billion Word and Enwiki8 datasets against strong baselines. In-depth ablation studies are performed to understand the mechanics of GNN-LM.

通过建立一个包含语料库中与输入内容相关的类语境环境节点的有向异构图，将GNN-LM扩展到传统的神经语言模型中（LM），并基于该图构建GNN网络来提高模型的泛化能力，有效地实现了对参考上下文的直接访问，展示了它在不同数据集上优于现有方法的结果（例如在WikiText-103上的最佳困惑度约为14.8）。

基于全局上下文的GNN语言建模