Evaluation Metrics is an important question for model evaluation and model selection in binary classification tasks. This study investigates how consistent metrics are at evaluating different models under different data scenarios. Analyzing over 150 data scenarios and 18 model evaluation metrics using statistical simulation, I find that for binary classification tasks, evaluation metrics that are less influenced by prevalence offer more consistent ranking of a set of different models. In particular, Area Under the ROC Curve (AUC) has smallest variance in ranking of different models. Matthew's correlation coefficient as a more strict measure of model performance has the second smallest variance. These patterns holds across a rich set of data scenarios and five commonly used machine learning models as well as a naive random guess model. The results have significant implications for model evaluation and model selection in binary classification tasks.

本研究针对二分类任务中的模型评估问题，探讨了不同数据场景下评估指标的一致性。通过对150多个数据场景和18种模型评估指标的统计模拟分析，发现ROC曲线下面积（AUC）在不同模型的排名中具有最小方差，提供了更一致的模型评估，这对模型选择具有重要影响。

ROC曲线下面积在二分类评估中具有最一致的评价