The assessment of binary classifier performance traditionally centers on
discriminative ability using metrics, such as accuracy. However, these metrics
often disregard the model's inherent uncertainty, especially when dealing with
sensitive decision-making domains, such as finance or healthcare. Given that
model-predicted scores are commonly seen as event probabilities, calibration is
crucial for accurate interpretation. In our study, we analyze the sensitivity
of various calibration measures to score distortions and introduce a refined
metric, the Local Calibration Score. Comparing recalibration methods, we
advocate for local regressions, emphasizing their dual role as effective
recalibration tools and facilitators of smoother visualizations. We apply these
findings in a real-world scenario using Random Forest classifier and regressor
to predict credit default while simultaneously measuring calibration during
performance optimization.

我们分析了各种校准度量对分数失真的敏感性，并引入了一种精确的度量标准，即本地校准分数，比较了校准方法，提倡使用局部回归，强调其作为有效校准工具和更平滑可视化的双重作用。我们在真实世界情景中应用这些发现，使用随机森林分类器和回归器预测信用违约，并在性能优化过程中同时测量校准度。