Knowledge of the domain of applicability of a machine learning model is
essential to ensuring accurate and reliable model predictions. In this work, we
develop a new approach of assessing model domain and demonstrate that our
approach provides accurate and meaningful designation of in-domain versus
out-of-domain when applied across multiple model types and material property
data sets. Our approach assesses the distance between a test and training data
point in feature space by using kernel density estimation and shows that this
distance provides an effective tool for domain determination. We show that
chemical groups considered unrelated based on established chemical knowledge
exhibit significant dissimilarities by our measure. We also show that high
measures of dissimilarity are associated with poor model performance (i.e.,
high residual magnitudes) and poor estimates of model uncertainty (i.e.,
unreliable uncertainty estimation). Automated tools are provided to enable
researchers to establish acceptable dissimilarity thresholds to identify
whether new predictions of their own machine learning models are in-domain
versus out-of-domain.

我们发展了一种新方法来评估模型的领域适用性，并展示了我们的方法在多个模型类型和材料性质数据集中提供准确和有意义的领域划分。我们的方法通过使用核密度估计计算特征空间中测试数据点与训练数据点的距离，并展示该距离为确定领域提供了有效的工具。我们展示了根据已有化学知识视为无关的化学组合在我们的测量中表现出显著的差异性。我们还展示了高度差异测量与模型性能不佳（即残差大小）和模型不确定性估计不可靠（即不可靠的不确定性估计）相关。我们提供了自动化工具，以帮助研究人员建立可接受的差异阈值，以确定他们自己的机器学习模型的新预测是领域内还是领域外。