Transformers have achieved remarkable performance in widespread fields,
including natural language processing, computer vision and graph mining.
However, vanilla Transformer architectures have not yielded promising
improvements in the Knowledge Graph (KG) representations, where the
translational distance paradigm dominates this area. Note that vanilla
Transformer architectures struggle to capture the intrinsically heterogeneous
structural and semantic information of knowledge graphs. To this end, we
propose a new variant of Transformer for knowledge graph representations dubbed
Relphormer. Specifically, we introduce Triple2Seq which can dynamically sample
contextualized sub-graph sequences as the input to alleviate the heterogeneity
issue. We propose a novel structure-enhanced self-attention mechanism to encode
the relational information and keep the semantic information within entities
and relations. Moreover, we utilize masked knowledge modeling for general
knowledge graph representation learning, which can be applied to various
KG-based tasks including knowledge graph completion, question answering, and
recommendation. Experimental results on six datasets show that Relphormer can
obtain better performance compared with baselines. Code is available in
this https URL.

本研究提出了一种新型变压器架构 Relphormer，其中引入 Triple2Seq 来动态采样上下文化子图序列作为输入以解决知识图谱中异构结构和语义信息难以捕获的问题，并提出了一种 novel self-attention 机制以编码关系信息。实验结果表明，相比基线模型，Relphormer 获得了更好的性能。