Molecular dynamic simulations are important in computational physics,
chemistry, material, and biology. Machine learning-based methods have shown
strong abilities in predicting molecular energy and properties and are much
faster than DFT calculations. Molecular energy is at least related to atoms,
bonds, bond angles, torsion angles, and nonbonding atom pairs. Previous
Transformer models only use atoms as inputs which lack explicit modeling of the
aforementioned factors. To alleviate this limitation, we propose Moleformer, a
novel Transformer architecture that takes nodes (atoms) and edges (bonds and
nonbonding atom pairs) as inputs and models the interactions among them using
rotational and translational invariant geometry-aware spatial encoding.
Proposed spatial encoding calculates relative position information including
distances and angles among nodes and edges. We benchmark Moleformer on OC20 and
QM9 datasets, and our model achieves state-of-the-art on the initial state to
relaxed energy prediction of OC20 and is very competitive in QM9 on predicting
quantum chemical properties compared to other Transformer and Graph Neural
Network (GNN) methods which proves the effectiveness of the proposed
geometry-aware spatial encoding in Moleformer.

我们提出了一种新颖的 Transformer 架构 Moleformer，它将节点（原子）和边（键和非键原子对）作为输入，并使用旋转不变性和平移不变性的几何感知空间编码来模拟它们之间的相互作用。我们在 OC20 和 QM9 数据集上进行了基准测试，Moleformer 在 OC20 的初始状态到松弛能量预测方面实现了最先进的水平，并且在预测量子化学性质方面与其他 Transformer 和图神经网络方法相比非常有竞争力，证明了所提出的几何感知空间编码在 Moleformer 中的有效性。