Knowledge-enhanced Pre-trained Language Model (PLM) has recently received
significant attention, which aims to incorporate factual knowledge into PLMs.
However, most existing methods modify the internal structures of fixed types of
PLMs by stacking complicated modules, and introduce redundant and irrelevant
factual knowledge from knowledge bases (KBs). In this paper, to address these
problems, we introduce a seminal knowledge prompting paradigm and further
propose a knowledge-prompting-based PLM framework KP-PLM. This framework can be
flexibly combined with existing mainstream PLMs. Specifically, we first
construct a knowledge sub-graph from KBs for each context. Then we design
multiple continuous prompts rules and transform the knowledge sub-graph into
natural language prompts. To further leverage the factual knowledge from these
prompts, we propose two novel knowledge-aware self-supervised tasks including
prompt relevance inspection and masked prompt modeling. Extensive experiments
on multiple natural language understanding (NLU) tasks show the superiority of
KP-PLM over other state-of-the-art methods in both full-resource and
low-resource settings.

本研究提出了知识提示范式和基于知识提示的 Pre-trained 语言模型 KP-PLM 框架，该框架通过多种连续提示规则将知识子图转化为自然语言提示，并在这些提示的基础上提出了两个新的知识感知无监督任务，实验证明了 KP-PLM 在多个自然语言理解任务中的优越性。