Most knowledge graphs (KGs) are incomplete, which motivates one important
research topic on automatically complementing knowledge graphs. However,
evaluation of knowledge graph completion (KGC) models often ignores the
incompleteness -- facts in the test set are ranked against all unknown triplets
which may contain a large number of missing facts not included in the KG yet.
Treating all unknown triplets as false is called the closed-world assumption.
This closed-world assumption might negatively affect the fairness and
consistency of the evaluation metrics. In this paper, we study KGC evaluation
under a more realistic setting, namely the open-world assumption, where unknown
triplets are considered to include many missing facts not included in the
training or test sets. For the currently most used metrics such as mean
reciprocal rank (MRR) and Hits@K, we point out that their behavior may be
unexpected under the open-world assumption. Specifically, with not many missing
facts, their numbers show a logarithmic trend with respect to the true strength
of the model, and thus, the metric increase could be insignificant in terms of
reflecting the true model improvement. Further, considering the variance, we
show that the degradation in the reported numbers may result in incorrect
comparisons between different models, where stronger models may have lower
metric numbers. We validate the phenomenon both theoretically and
experimentally. Finally, we suggest possible causes and solutions for this
problem. Our code and data are available at
this https URL .

本文研究了在更现实的开放世界假设下，知识图谱补全模型的评估，指出在测试时采用的指标可能会不正确反映模型的表现，因为未知的三元组可能包含许多缺失的事实，不在训练或测试集中，因此建议对数据进行更多的探索和量化分析以改进现有的评估方法。