Despite their widespread success in various domains, Transformer networks
have yet to perform well across datasets in the domain of 3D atomistic graphs
such as molecules even when 3D-related inductive biases like translational
invariance and rotational equivariance are considered. In this paper, we
demonstrate that Transformers can generalize well to 3D atomistic graphs and
present Equiformer, a graph neural network leveraging the strength of
Transformer architectures and incorporating SE(3)/E(3)-equivariant features
based on irreducible representations (irreps). First, we propose a simple and
effective architecture by only replacing original operations in Transformers
with their equivariant counterparts and including tensor products. Using
equivariant operations enables encoding equivariant information in channels of
irreps features without complicating graph structures. With minimal
modifications to Transformers, this architecture has already achieved strong
empirical results. Second, we propose a novel attention mechanism called
equivariant graph attention, which improves upon typical attention in
Transformers through replacing dot product attention with multi-layer
perceptron attention and including non-linear message passing. With these two
innovations, Equiformer achieves competitive results to previous models on QM9,
MD17 and OC20 datasets.

本文提出的 Equiformer 是一种图神经网络，利用变换器架构的优势，并结合基于不可约表示（irreps）的 SE (3)/E (3) 等变特征，取得了在 QM9、MD17 和 OC20 数据集上与之前模型相当的好的性能。