Matbench Discovery simulates the deployment of machine learning (ML) energy
models in a high-throughput search for stable inorganic crystals. We address
the disconnect between (i) thermodynamic stability and formation energy and
(ii) in-domain vs out-of-distribution performance. Alongside this paper, we
publish a Python package to aid with future model submissions and a growing
online leaderboard with further insights into trade-offs between various
performance metrics. To answer the question which ML methodology performs best
at materials discovery, our initial release explores a variety of models
including random forests, graph neural networks (GNN), one-shot predictors,
iterative Bayesian optimizers and universal interatomic potentials (UIP).
Ranked best-to-worst by their test set F1 score on thermodynamic stability
prediction, we find CHGNet > M3GNet > MACE > ALIGNN > MEGNet > CGCNN > CGCNN+P
> Wrenformer > BOWSR > Voronoi tessellation fingerprints with random forest.
The top 3 models are UIPs, the winning methodology for ML-guided materials
discovery, achieving F1 scores of ~0.6 for crystal stability classification and
discovery acceleration factors (DAF) of up to 5x on the first 10k most stable
predictions compared to dummy selection from our test set. We also highlight a
sharp disconnect between commonly used global regression metrics and more
task-relevant classification metrics. Accurate regressors are susceptible to
unexpectedly high false-positive rates if those accurate predictions lie close
to the decision boundary at 0 eV/atom above the convex hull where most
materials are. Our results highlight the need to focus on classification
metrics that actually correlate with improved stability hit rate.

Matbench Discovery 通过部署机器学习（ML）能源模型在高通量搜索稳定无机晶体中模拟。我们解决了（i）热力学稳定性与形成能之间的脱节和（ii）域内与域外的性能之间的差异问题。通过本文和我们的 Python 包以及在线排行榜，我们探索了各个性能指标之间的权衡，回答了在材料发现方面哪种 ML 方法表现最佳的问题。我们发现在热力学稳定性预测的测试集 F1 得分中，CHGNet > M3GNet > MACE > ALIGNN > MEGNet > CGCNN > CGCNN+P > Wrenformer > BOWSR > Voronoi tessellation fingerprints with random forest。前三个模型是 UIPs，是 ML 引导材料发现的获胜方法，其晶体稳定性分类的 F1 得分达到了约 0.6，并且在与我们的测试集的前 10000 个最稳定预测的虚拟选择相比，发现加速因子（DAF）最高可达 5 倍。我们还强调了常用的全局回归指标与更具任务相关性的分类指标之间的明显差异。准确的回归器如果那些准确的预测接近 0 eV / 原子的凸包上的决策边界，那么它们就容易出现意外高的假阳性率，而大多数材料都是接近该边界的。我们的结果突显了需要关注与提高稳定性命中率实际相关的分类指标。