Question Answering over Knowledge Graph (KGQA) aims to seek answer entities
for the natural language question from a large-scale Knowledge Graph~(KG). To
better perform reasoning on KG, recent work typically adopts a pre-trained
language model~(PLM) to model the question, and a graph neural network~(GNN)
based module to perform multi-hop reasoning on the KG. Despite the
effectiveness, due to the divergence in model architecture, the PLM and GNN are
not closely integrated, limiting the knowledge sharing and fine-grained feature
interactions. To solve it, we aim to simplify the above two-module approach,
and develop a more capable PLM that can directly support subgraph reasoning for
KGQA, namely ReasoningLM. In our approach, we propose a subgraph-aware
self-attention mechanism to imitate the GNN for performing structured
reasoning, and also adopt an adaptation tuning strategy to adapt the model
parameters with 20,000 subgraphs with synthesized questions. After adaptation,
the PLM can be parameter-efficient fine-tuned on downstream tasks. Experiments
show that ReasoningLM surpasses state-of-the-art models by a large margin, even
with fewer updated parameters and less training data. Our codes and data are
publicly available at~https://github.com/RUCAIBox/ReasoningLM.

Question Answering over Knowledge Graph (KGQA) aims to seek answer entities for the natural language question from a large-scale Knowledge Graph. To better perform reasoning on KG, recent work typically adopts a pre-trained language model (PLM) and a graph neural network (GNN) module, but these are not closely integrated. This paper proposes ReasoningLM, a more capable PLM that directly supports subgraph reasoning for KGQA, outperforming state-of-the-art models.