Pretraining a language model (LM) on text has been shown to help various
downstream NLP tasks. Recent works show that a knowledge graph (KG) can
complement text data, offering structured background knowledge that provides a
useful scaffold for reasoning. However, these works are not pretrained to learn
a deep fusion of the two modalities at scale, limiting the potential to acquire
fully joint representations of text and KG. Here we propose DRAGON (Deep
Bidirectional Language-Knowledge Graph Pretraining), a self-supervised approach
to pretraining a deeply joint language-knowledge foundation model from text and
KG at scale. Specifically, our model takes pairs of text segments and relevant
KG subgraphs as input and bidirectionally fuses information from both
modalities. We pretrain this model by unifying two self-supervised reasoning
tasks, masked language modeling and KG link prediction. DRAGON outperforms
existing LM and LM+KG models on diverse downstream tasks including question
answering across general and biomedical domains, with +5% absolute gain on
average. In particular, DRAGON achieves notable performance on complex
reasoning about language and knowledge (+10% on questions involving long
contexts or multi-step reasoning) and low-resource QA (+8% on OBQA and
RiddleSense), and new state-of-the-art results on various BioNLP tasks. Our
code and trained models are available at
this https URL.

用自我监督的方法，从文本和知识图谱中预先训练出深度语言 - 知识图谱联合模型 DRAGON，而 DRAGON 在一系列下游任务中实现了良好性能，尤其是在语言和知识复杂推理方面的优异表现，以及在各种 BioNLP 任务中的最新成果。