Matbench Discovery simulates the deployment of machine learning (ML) energy
models in a high-throughput search for stable inorganic crystals. We address
the disconnect between (i) thermodynamic stability and formation energy and
(ii) in-domain vs out-of-distribution performance. Alongside this paper, we
publish a Python package to aid with future model submissions and a growing
online leaderboard with further insights into trade-offs between various
performance metrics. To answer the question which ML methodology performs best
at materials discovery, our initial release explores a variety of models
including random forests, graph neural networks (GNN), one-shot predictors,
iterative Bayesian optimizers and universal interatomic potentials (UIP).
Ranked best-to-worst by their test set F1 score on thermodynamic stability
prediction, we find CHGNet > M3GNet > MACE > ALIGNN > MEGNet > CGCNN > CGCNN+P
> Wrenformer > BOWSR > Voronoi tessellation fingerprints with random forest.
The top 3 models are UIPs, the winning methodology for ML-guided materials
discovery, achieving F1 scores of ~0.6 for crystal stability classification and
discovery acceleration factors (DAF) of up to 5x on the first 10k most stable
predictions compared to dummy selection from our test set. We also highlight a
sharp disconnect between commonly used global regression metrics and more
task-relevant classification metrics. Accurate regressors are susceptible to
unexpectedly high false-positive rates if those accurate predictions lie close
to the decision boundary at 0 eV/atom above the convex hull where most
materials are. Our results highlight the need to focus on classification
metrics that actually correlate with improved stability hit rate.

Matbench Discovery 通过部署机器学习（ML）能源模型在高通量搜索稳定无机晶体中模拟。我们解决了（i）热力学稳定性与形成能之间的脱节和（ii）域内与域外的性能之间的差异问题。通过本文和我们的 Python 包以及在线排行榜，我们探索了各个性能指标之间的权衡，回答了在材料发现方面哪种 ML 方法表现最佳的问题。我们发现在热力学稳定性预测的测试集 F1 得分中，CHGNet > M3GNet > MACE > ALIGNN > MEGNet > CGCNN > CGCNN+P > Wrenformer > BOWSR > Voronoi tessellation fingerprints with random forest。前三个模型是 UIPs，是 ML 引导材料发现的获胜方法，其晶体稳定性分类的 F1 得分达到了约 0.6，并且在与我们的测试集的前 10000 个最稳定预测的虚拟选择相比，发现加速因子（DAF）最高可达 5 倍。我们还强调了常用的全局回归指标与更具任务相关性的分类指标之间的明显差异。准确的回归器如果那些准确的预测接近 0 eV / 原子的凸包上的决策边界，那么它们就容易出现意外高的假阳性率，而大多数材料都是接近该边界的。我们的结果突显了需要关注与提高稳定性命中率实际相关的分类指标。

Matbench Discovery  - 机器学习晶体稳定性预测的评估框架

Matbench Discovery -- An evaluation framework for machine learning  crystal stability prediction

The recently proposed crystal graph convolutional neural network (CGCNN)
offers a highly versatile and accurate machine learning (ML) framework by
learning material properties directly from graph-like representations of
crystal structures ("crystal graphs"). Here, we develop an improved variant of
the CGCNN model (iCGCNN) that outperforms the original by incorporating
information of the Voronoi tessellated crystal structure, explicit 3-body
correlations of neighboring constituent atoms, and an optimized chemical
representation of interatomic bonds in the crystal graphs. We demonstrate the
accuracy of the improved framework in two distinct illustrations: First, when
trained/validated on 180,000/20,000 density functional theory (DFT) calculated
thermodynamic stability entries taken from the Open Quantum Materials Database
(OQMD) and evaluated on a separate test set of 230,000 entries, iCGCNN achieves
a predictive accuracy that is significantly improved, i.e., 20% higher than
that of the original CGCNN. Second, when used to assist high-throughput search
for materials in the ThCr2Si2 structure-type, iCGCNN exhibited a success rate
of 31% which is 310 times higher than an undirected high-throughput search and
2.4 times higher than that of the original CGCNN. Using both CGCNN and iCGCNN,
we screened 132,600 compounds with elemental decorations of the ThCr2Si2
prototype crystal structure and identified a total of 97 new unique stable
compounds by performing 757 DFT calculations, accelerating the computational
time of the high-throughput search by a factor of 130. Our results suggest that
the iCGCNN can be used to accelerate high-throughput discoveries of new
materials by quickly and accurately identifying crystalline compounds with
properties of interest.

本文中提出了一种改进的晶体图卷积神经网络（iCGCNN）来学习晶体结构的材料属性，该方法不仅采用 Voronoi 剖分晶体结构，还考虑了邻近组分原子的 3 体相互作用和晶体图中原子间化学键的优化表示。研究表明，iCGCNN 方法可以用于通过快速准确地识别具有所需性质的晶体化合物来加速高通量新材料发现，从而显著提高预测准确性和成功率。