The reliability of a machine learning model's confidence in its predictions is critical for highrisk applications. Calibration-the idea that a model's predicted probabilities of outcomes reflect true probabilities of those outcomes-formalizes this notion. While analyzing the calibration of deep neural networks, we've identified core problems with the way calibration is currently measured. We design the Thresholded Adaptive Calibration Error (TACE) metric to resolve these pathologies and show that it outperforms other metrics, especially in settings where predictions beyond the maximum prediction that is chosen as the output class matter. There are many cases where what a practitioner cares about is the calibration of a specific prediction, and so we introduce a dynamic programming based Prediction Specific Calibration Error (PSCE) that smoothly considers the calibration of nearby predictions to give an estimate of the calibration error of a specific prediction.

本文探讨了机器学习分类器中出现的过度置信和欠置信问题，介绍了衡量分类器置信度的度量标准——准确率校准度，以及该度量标准不足的缺点；进而进行了一项全面的实证研究，并开源了一种度量方法的库。实证研究发现，不同的度量标准对于分类器校准度的优化与评价可能会带来截然不同的结论。

深度学习中的校准度量