Designing a single neural network architecture that performs competitively
across a range of molecule property prediction tasks remains largely an open
challenge, and its solution may unlock a widespread use of deep learning in the
drug discovery industry. To move towards this goal, we propose Molecule
Attention Transformer (MAT). Our key innovation is to augment the attention
mechanism in Transformer using inter-atomic distances and the molecular graph
structure. Experiments show that MAT performs competitively on a diverse set of
molecular prediction tasks. Most importantly, with a simple self-supervised
pretraining, MAT requires tuning of only a few hyperparameter values to achieve
state-of-the-art performance on downstream tasks. Finally, we show that
attention weights learned by MAT are interpretable from the chemical point of
view.

通过在 Transformer 中使用原子间距离和分子结构来扩充注意力机制，我们提出了 Molecule Attention Transformer（MAT）神经网络模型，其可以在一系列不同的分子预测任务中以竞争优势表现。并且，我们展示了一个简单的自监督预训练可以让 MAT 以很少的超参数调整就达到最新技术水平，并且 MAT 学习到的注意力权重可以从化学角度解释。