Recent advances in graph neural networks (GNNs) have allowed molecular
simulations with accuracy on par with conventional gold-standard methods at a
fraction of the computational cost. Nonetheless, as the field has been
progressing to bigger and more complex architectures, state-of-the-art GNNs
have become largely prohibitive for many large-scale applications. In this
paper, we, for the first time, explore the utility of knowledge distillation
(KD) for accelerating molecular GNNs. To this end, we devise KD strategies that
facilitate the distillation of hidden representations in directional and
equivariant GNNs and evaluate their performance on the regression task of
energy and force prediction. We validate our protocols across different
teacher-student configurations and demonstrate that they can boost the
predictive accuracy of student models without altering their architecture. We
also conduct comprehensive optimization of various components of our framework,
and investigate the potential of data augmentation to further enhance
performance. All in all, we manage to close as much as 59% of the gap in
predictive accuracy between models like GemNet-OC and PaiNN with zero
additional cost at inference.

本文探讨了使用知识蒸馏加速分子图神经网络的实用性，研究结果显示可以无需改变体系结构就能提高预测准确性达到近 60%。

知识蒸馏加速分子图神经网络

Accelerating Molecular Graph Neural Networks via Knowledge Distillation

Pretraining molecular representation models without labels is fundamental to
various applications. Conventional methods mainly process 2D molecular graphs
and focus solely on 2D tasks, making their pretrained models incapable of
characterizing 3D geometry and thus defective for downstream 3D tasks. In this
work, we tackle 3D molecular pretraining in a complete and novel sense. In
particular, we first propose to adopt an equivariant energy-based model as the
backbone for pretraining, which enjoys the merits of fulfilling the symmetry of
3D space. Then we develop a node-level pretraining loss for force prediction,
where we further exploit the Riemann-Gaussian distribution to ensure the loss
to be E(3)-invariant, enabling more robustness. Moreover, a graph-level noise
scale prediction task is also leveraged to further promote the eventual
performance. We evaluate our model pretrained from a large-scale 3D dataset
GEOM-QM9 on two challenging 3D benchmarks: MD17 and QM9. Experimental results
demonstrate the efficacy of our method against current state-of-the-art
pretraining approaches, and verify the validity of our design for each proposed
component.

本文提出一种新的三维分子预训练方法，该方法采用等变能量基模型作为主干网络，以预测节点力信息为目标进行节点级预训练损失计算，并采用高斯分布以保证其 E（3）不变性，同时通过图层级噪声缩放预测任务来进一步提高模型性能，实验结果表明，该方法在两个挑战性的三维基准测试上均取得了比当前最先进的预训练方法更佳的表现。