Equivariant Transformers such as Equiformer have demonstrated the efficacy of applying Transformers to the domain of 3D atomistic systems. However, they are still limited to small degrees of equivariant representations due to their computational complexity. In this paper, we investigate whether these architectures can scale well to higher degrees. Starting from Equiformer, we first replace $SO(3)$ convolutions with eSCN convolutions to efficiently incorporate higher-degree tensors. Then, to better leverage the power of higher degrees, we propose three architectural improvements -- attention re-normalization, separable $S^2$ activation and separable layer normalization. Putting this all together, we propose EquiformerV2, which outperforms previous state-of-the-art methods on the large-scale OC20 dataset by up to $12\%$ on forces, $4\%$ on energies, offers better speed-accuracy trade-offs, and $2\times$ reduction in DFT calculations needed for computing adsorption energies.

本研究提出 EquiformerV2 基于 eSCN 卷积和三个结构改进在大规模 OC20 数据集上取得了比之前方法高达 12% 的力量和 4% 的能量数据性能，并在速度-精度平衡和计算吸附能的 DFT 计算方面实现了 2 倍的降低。

EquiformerV2：用于扩展更高程度表示的改进等变Transformer