Incorporating factual knowledge into pre-trained language models (PLM) such
as BERT is an emerging trend in recent NLP studies. However, most of the
existing methods combine the external knowledge integration module with a
modified pre-training loss and re-implement the pre-training process on the
large-scale corpus. Re-pretraining these models is usually resource-consuming,
and difficult to adapt to another domain with a different knowledge graph (KG).
Besides, those works either cannot embed knowledge context dynamically
according to textual context or struggle with the knowledge ambiguity issue. In
this paper, we propose a novel knowledge-aware language model framework based
on fine-tuning process, which equips PLM with a unified knowledge-enhanced text
graph that contains both text and multi-relational sub-graphs extracted from
KG. We design a hierarchical relational-graph-based message passing mechanism,
which can allow the representations of injected KG and text to mutually update
each other and can dynamically select ambiguous mentioned entities that share
the same text. Our empirical results show that our model can efficiently
incorporate world knowledge from KGs into existing language models such as
BERT, and achieve significant improvement on the machine reading comprehension
(MRC) task compared with other knowledge-enhanced models.

该论文提出了一种基于微调过程的知识感知语言模型框架，将领域知识通过关系图嵌入到预训练语言模型中并进行动态更新，取得了比其他知识增强模型更显著的机器阅读理解任务的性能改进。