A machine learning model is calibrated if its predicted probability for an
outcome matches the observed frequency for that outcome conditional on the
model prediction. This property has become increasingly important as the impact
of machine learning models has continued to spread to various domains. As a
result, there are now a dizzying number of recent papers on measuring and
improving the calibration of (specifically deep learning) models. In this work,
we reassess the reporting of calibration metrics in the recent literature. We
show that there exist trivial recalibration approaches that can appear
seemingly state-of-the-art unless calibration and prediction metrics (i.e. test
accuracy) are accompanied by additional generalization metrics such as negative
log-likelihood. We then derive a calibration-based decomposition of Bregman
divergences that can be used to both motivate a choice of calibration metric
based on a generalization metric, and to detect trivial calibration. Finally,
we apply these ideas to develop a new extension to reliability diagrams that
can be used to jointly visualize calibration as well as the estimated
generalization error of a model.

机器学习模型的校准至关重要，本文重新评估了最近文献中校准指标的报告情况，并提出了一种基于广义指标的校准度量选择和检测方法以及一种新的可视化校准和模型估计广义误差的方法。

重新评估机器学习模型的校准比较和改进方法

Reassessing How to Compare and Improve the Calibration of Machine  Learning Models

The search for effective and robust metrics has been the focus of recent
theoretical and empirical work on generalization of deep neural networks (NNs).
In this paper, we discuss the performance of natural language processing (NLP)
models, and we evaluate various existing and novel generalization metrics.
Compared to prior studies, we (i) focus on NLP instead of computer vision (CV),
(ii) focus on generalization metrics that predict test error instead of the
generalization gap, (iii) focus on generalization metrics that do not need the
access to data, and (iv) focus on the heavy-tail (HT) phenomenon that has
received comparatively less attention in the study of NNs. We extend recent
HT-based work which focuses on power law (PL) distributions, and we study
exponential and exponentially truncated power law (E-TPL) fitting to the
empirical spectral densities (ESDs) of weight matrices. Our empirical studies
are carried on (i) hundreds of Transformers trained in different settings, in
which we systematically vary different hyperparameters, (ii) a total of 51
pretrained Transformers from eight families of Huggingface NLP models,
including BERT, GPT2, etc., and (iii) a total of 28 existing and novel
generalization metrics. From our empirical analyses, we show that shape
metrics, or the metrics obtained from fitting the shape of the ESDs, perform
uniformly better at predicting generalization performance than scale metrics
commonly studied in the literature, as measured by the rank correlations with
the generalization performance. We also show that among the three HT
distributions considered in our paper, the E-TPL fitting of ESDs performs the
most robustly when the models are trained in experimental settings, while the
PL fitting achieves the best performance on well-trained Huggingface models,
and that both E-TPL and PL metrics (which are both shape metrics) outperform
scale metrics.

本文研究了自然语言处理模型中，关于深度神经网络的推广问题，提出了几种新的推广度量标准并作了实证研究。研究表明，通过拟合嵌入矩阵的形状的度量标准与与推广表现有着更强的相关性。