Question Answering over Knowledge Graph (KGQA) aims to seek answer entities
for the natural language question from a large-scale Knowledge Graph~(KG). To
better perform reasoning on KG, recent work typically adopts a pre-trained
language model~(PLM) to model the question, and a graph neural network~(GNN)
based module to perform multi-hop reasoning on the KG. Despite the
effectiveness, due to the divergence in model architecture, the PLM and GNN are
not closely integrated, limiting the knowledge sharing and fine-grained feature
interactions. To solve it, we aim to simplify the above two-module approach,
and develop a more capable PLM that can directly support subgraph reasoning for
KGQA, namely ReasoningLM. In our approach, we propose a subgraph-aware
self-attention mechanism to imitate the GNN for performing structured
reasoning, and also adopt an adaptation tuning strategy to adapt the model
parameters with 20,000 subgraphs with synthesized questions. After adaptation,
the PLM can be parameter-efficient fine-tuned on downstream tasks. Experiments
show that ReasoningLM surpasses state-of-the-art models by a large margin, even
with fewer updated parameters and less training data. Our codes and data are
publicly available at~https://github.com/RUCAIBox/ReasoningLM.

Question Answering over Knowledge Graph (KGQA) aims to seek answer entities for the natural language question from a large-scale Knowledge Graph. To better perform reasoning on KG, recent work typically adopts a pre-trained language model (PLM) and a graph neural network (GNN) module, but these are not closely integrated. This paper proposes ReasoningLM, a more capable PLM that directly supports subgraph reasoning for KGQA, outperforming state-of-the-art models.

ReasoningLM: 在预训练语言模型中实现知识图谱中的结构子图推理以解决问题回答

ReasoningLM: Enabling Structural Subgraph Reasoning in Pre-trained  Language Models for Question Answering over Knowledge Graph

In knowledge graph completion (KGC), predicting triples involving emerging
entities and/or relations, which are unseen when the KG embeddings are learned,
has become a critical challenge. Subgraph reasoning with message passing is a
promising and popular solution. Some recent methods have achieved good
performance, but they (i) usually can only predict triples involving unseen
entities alone, failing to address more realistic fully inductive situations
with both unseen entities and unseen relations, and (ii) often conduct message
passing over the entities with the relation patterns not fully utilized. In
this study, we propose a new method named RMPI which uses a novel Relational
Message Passing network for fully Inductive KGC. It passes messages directly
between relations to make full use of the relation patterns for subgraph
reasoning with new techniques on graph transformation, graph pruning,
relation-aware neighborhood attention, addressing empty subgraphs, etc., and
can utilize the relation semantics defined in the ontological schema of KG.
Extensive evaluation on multiple benchmarks has shown the effectiveness of
techniques involved in RMPI and its better performance compared with the
existing methods that support fully inductive KGC. RMPI is also comparable to
the state-of-the-art partially inductive KGC methods with very promising
results achieved. Our codes and data are available at
this https URL.

本研究提出了一种新方法 RMPI，它使用关系信息直接传递消息来充分利用子图推理中的关系模式，具有一定的归纳能力，并且可以比现有的全归纳 KGC 方法表现更好。