Pre-trained language representation models, such as BERT, capture a general
language representation from large-scale corpora, but lack domain-specific
knowledge. When reading a domain text, experts make inferences with relevant
knowledge. For machines to achieve this capability, we propose a
knowledge-enabled language representation model (K-BERT) with knowledge graphs
(KGs), in which triples are injected into the sentences as domain knowledge.
However, too much knowledge incorporation may divert the sentence from its
correct meaning, which is called knowledge noise (KN) issue. To overcome KN,
K-BERT introduces soft-position and visible matrix to limit the impact of
knowledge. K-BERT can easily inject domain knowledge into the models by
equipped with a KG without pre-training by-self because it is capable of
loading model parameters from the pre-trained BERT. Our investigation reveals
promising results in twelve NLP tasks. Especially in domain-specific tasks
(including finance, law, and medicine), K-BERT significantly outperforms BERT,
which demonstrates that K-BERT is an excellent choice for solving the
knowledge-driven problems that require experts.

提出了一种基于知识图谱的语言表示模型（K-BERT），可以将领域特定的知识注入到模型中，以用于解决需要专家知识的领域特定问题，实验证明其在 12 个 NLP 任务中有很好的表现。