Despite recent successes in natural language processing and computer vision,
Transformer suffers from the scalability problem when dealing with graphs. The
computational complexity is unacceptable for large-scale graphs, e.g.,
knowledge graphs. One solution is to consider only the near neighbors, which,
however, will lose the key merit of Transformer to attend to the elements at
any distance. In this paper, we propose a new Transformer architecture, named
dual-encoding Transformer (DET). DET has a structural encoder to aggregate
information from connected neighbors and a semantic encoder to focus on
semantically useful distant nodes. In comparison with resorting to multi-hop
neighbors, DET seeks the desired distant neighbors via self-supervised
training. We further find these two encoders can be incorporated to boost each
others' performance. Our experiments demonstrate DET has achieved superior
performance compared to the respective state-of-the-art methods in dealing with
molecules, networks and knowledge graphs with various sizes.

本研究提出了一种名为 DET 的新型 Transformer 架构，采用结构编码器聚合连接邻居的信息和语义编码器聚焦于远距离节点，通过自监督训练寻找所需的远距离邻居，比使用多跳邻居的方法更为优越。实验结果表明，DET 在处理分子、网络和各种规模的知识图谱方面比对应的最新方法具有更优越的性能表现。