Applications such as weather forecasting and personalized medicine demand
models that output calibrated probability estimates---those representative of
the true likelihood of a prediction. Most models are not calibrated out of the
box but are recalibrated by post-processing model outputs. We find in this work
that popular recalibration methods like Platt scaling and temperature scaling
are (i) less calibrated than reported, and (ii) current techniques cannot
estimate how miscalibrated they are. An alternative method, histogram binning,
has measurable calibration error but is sample inefficient---it requires
$O(B/\epsilon^2)$ samples, compared to $O(1/\epsilon^2)$ for scaling methods,
where $B$ is the number of distinct probabilities the model can output. To get
the best of both worlds, we introduce the scaling-binning calibrator, which
first fits a parametric function to reduce variance and then bins the function
values to actually ensure calibration. This requires only $O(1/\epsilon^2 + B)$
samples. Next, we show that we can estimate a model's calibration error more
accurately using an estimator from the meteorological community---or
equivalently measure its calibration error with fewer samples ($O(\sqrt{B})$
instead of $O(B)$). We validate our approach with multiclass calibration
experiments on CIFAR-10 and ImageNet, where we obtain a 35% lower calibration
error than histogram binning and, unlike scaling methods, guarantees on true
calibration. In these experiments, we also estimate the calibration error and
ECE more accurately than the commonly used plugin estimators. We implement all
these methods in a Python library:
this https URL

本篇论文提出一种基于 Platti scaling 和 temperature scaling 的直方图分块校准法，用于计算模型校准误差，并使用气象学中的估算器来更加准确地估计校准误差。作者在多类别校准实验中将缔合法的校准误差降低了 35％。