Knowledge-enhanced pre-trained models for language representation have been
shown to be more effective in knowledge base construction tasks (i.e.,~relation
extraction) than language models such as BERT. These knowledge-enhanced
language models incorporate knowledge into pre-training to generate
representations of entities or relationships. However, existing methods
typically represent each entity with a separate embedding. As a result, these
methods struggle to represent out-of-vocabulary entities and a large amount of
parameters, on top of their underlying token models (i.e.,~the transformer),
must be used and the number of entities that can be handled is limited in
practice due to memory constraints. Moreover, existing models still struggle to
represent entities and relationships simultaneously. To address these problems,
we propose a new pre-trained model that learns representations of both entities
and relationships from token spans and span pairs in the text respectively. By
encoding spans efficiently with span modules, our model can represent both
entities and their relationships but requires fewer parameters than existing
models. We pre-trained our model with the knowledge graph extracted from
Wikipedia and test it on a broad range of supervised and unsupervised
information extraction tasks. Results show that our model learns better
representations for both entities and relationships than baselines, while in
supervised settings, fine-tuning our model outperforms RoBERTa consistently and
achieves competitive results on information extraction tasks.

我们提出了一种新的预训练模型，利用基于维基百科的知识图谱实现了在实体和关系上的双重表示，以及利用跨度模块进行有效编码，相较于现有模型使用更少的参数。实验结果表明，相比 RoBERTa 模型，我们模型在信息抽取任务上表现更好，尤其在监督学习任务中达到了很高的竞争水平。