Knowledge-enhanced language representation learning has shown promising
results across various knowledge-intensive NLP tasks. However, prior methods
are limited in efficient utilization of multilingual knowledge graph (KG) data
for language model (LM) pretraining. They often train LMs with KGs in indirect
ways, relying on extra entity/relation embeddings to facilitate knowledge
injection. In this work, we explore methods to make better use of the
multilingual annotation and language agnostic property of KG triples, and
present novel knowledge based multilingual language models (KMLMs) trained
directly on the knowledge triples. We first generate a large amount of
multilingual synthetic sentences using the Wikidata KG triples. Then based on
the intra- and inter-sentence structures of the generated data, we design
pretraining tasks to enable the LMs to not only memorize the factual knowledge
but also learn useful logical patterns. Our pretrained KMLMs demonstrate
significant performance improvements on a wide range of knowledge-intensive
cross-lingual tasks, including named entity recognition (NER), factual
knowledge retrieval, relation classification, and a newly designed logical
reasoning task.

本文提出了使用多语言知识图谱三元组进行直接训练的基于知识的多语言语言模型。通过生成大量的多语言合成句子并设计预训练任务，实现了 KMLMs 对事实知识和逻辑模式的学习，并在跨语言任务中展示了显著的性能提升，包括命名实体识别（NER）、事实知识检索、关系分类和新设计的逻辑推理任务。