We show that standard Transformers without graph-specific modifications can
lead to promising results in graph learning both in theory and practice. Given
a graph, we simply treat all nodes and edges as independent tokens, augment
them with token embeddings, and feed them to a Transformer. With an appropriate
choice of token embeddings, we prove that this approach is theoretically at
least as expressive as an invariant graph network (2-IGN) composed of
equivariant linear layers, which is already more expressive than all
message-passing Graph Neural Networks (GNN). When trained on a large-scale
graph dataset (PCQM4Mv2), our method coined Tokenized Graph Transformer
(TokenGT) achieves significantly better results compared to GNN baselines and
competitive results compared to Transformer variants with sophisticated
graph-specific inductive bias. Our implementation is available at
this https URL.

本文提出了一种名为 Tokenized Graph Transformer（TokenGT）的新颖神经网络，它可以用于大规模图形学习，通过节点和边的独立表示进行转换，并且通过合适的 token 嵌入，TokenGT 可以在理论上至少与由等变线性层组成的不变图网络（2-IGN）一样有潜力进行表现，实践中，TokenGT 在大规模图形数据集（PCQM4Mv2）上的表现不仅优于基于消息传递的图神经网络（GNN）基线，而且与具有复杂图特定归纳偏差的变型 Transformer 相比也具有竞争力。