Transformers, adapted from natural language processing, are emerging as a
leading approach for graph representation learning. Contemporary graph
transformers often treat nodes or edges as separate tokens. This approach leads
to computational challenges for even moderately-sized graphs due to the
quadratic scaling of self-attention complexity with token count. In this paper,
we introduce SubFormer, a graph transformer that operates on subgraphs that
aggregate information by a message-passing mechanism. This approach reduces the
number of tokens and enhances learning long-range interactions. We demonstrate
SubFormer on benchmarks for predicting molecular properties from chemical
structures and show that it is competitive with state-of-the-art graph
transformers at a fraction of the computational cost, with training times on
the order of minutes on a consumer-grade graphics card. We interpret the
attention weights in terms of chemical structures. We show that SubFormer
exhibits limited over-smoothing and avoids over-squashing, which is prevalent
in traditional graph neural networks.

SubFormer 是一种图变换器，通过消息传递机制对子图进行操作，减少了标记数量，增强了学习长距离交互的能力。在预测分子属性和化学结构方面，SubFormer 与最先进的图变换器相比，在计算成本的一小部分情况下保持着竞争力，并且在消费级显卡上进行训练的时间仅为几分钟。我们通过对化学结构的注意权重进行解读，展示了 SubFormer 表现出有限的过度平滑和避免了传统图神经网络常见的过度压缩问题。

Transformers 高效分层化化学图学习器

Transformers are efficient hierarchical chemical graph learners

Transformers have shown improved performance when compared to previous
architectures for sequence processing such as RNNs. Despite their sizeable
performance gains, as recently suggested, the model is computationally
expensive to train and with a high parameter budget. In light of this, we
explore parameter-sharing methods in Transformers with a specific focus on
generative models. We perform an analysis of different parameter
sharing/reduction methods and develop the Subformer. Our model combines
sandwich-style parameter sharing, which overcomes naive cross-layer parameter
sharing in generative models, and self-attentive embedding factorization
(SAFE). Experiments on machine translation, abstractive summarization and
language modeling show that the Subformer can outperform the Transformer even
when using significantly fewer parameters.

介绍了一种在 Transformers 中使用参数共享方法的算法 Subformer，该算法通过结合几种不同的参数共享 / 降维方法，以及使用 SELF 进行自我关注的嵌入分解技术，以在机器翻译、提取式摘要和语言建模等任务中提高性能。