Research has shown that deep networks tend to be overly optimistic about
their predictions, leading to an underestimation of prediction errors. Due to
the limited nature of data, existing studies have proposed various methods
based on model prediction probabilities to bin the data and evaluate
calibration error. We propose a more generalized definition of calibration
error called Partitioned Calibration Error (PCE), revealing that the key
difference among these calibration error metrics lies in how the data space is
partitioned. We put forth an intuitive proposition that an accurate model
should be calibrated across any partition, suggesting that the input space
partitioning can extend beyond just the partitioning of prediction
probabilities, and include partitions directly related to the input. Through
semantic-related partitioning functions, we demonstrate that the relationship
between model accuracy and calibration lies in the granularity of the
partitioning function. This highlights the importance of partitioning criteria
for training a calibrated and accurate model. To validate the aforementioned
analysis, we propose a method that involves jointly learning a semantic aware
grouping function based on deep model features and logits to partition the data
space into subsets. Subsequently, a separate calibration function is learned
for each subset. Experimental results demonstrate that our approach achieves
significant performance improvements across multiple datasets and network
architectures, thus highlighting the importance of the partitioning function
for calibration.

文章提出了一种更广义的标定误差度量定义，即划分校准误差（Partitioned Calibration Error - PCE），并探讨了如何对数据空间进行划分，以使得深度神经网络可以更准确地校准预测。作者提出了通过语义相关的划分函数来划分输入空间的想法，通过实验证明该方法在多个数据集和网络架构上都可以获得显著的性能提升，突显了划分标准对于训练校准和准确的深度学习模型的重要性。