Deep neural networks are increasingly utilized in various machine learning tasks. However, as these models grow in complexity, they often face calibration issues, despite enhanced prediction accuracy. Many studies have endeavored to improve calibration performance through data preprocessing, the use of specific loss functions, and training frameworks. Yet, investigations into calibration properties have been somewhat overlooked. Our study leverages the Neural Architecture Search (NAS) search space, offering an exhaustive model architecture space for thorough calibration properties exploration. We specifically create a model calibration dataset. This dataset evaluates 90 bin-based and 12 additional calibration measurements across 117,702 unique neural networks within the widely employed NATS-Bench search space. Our analysis aims to answer several longstanding questions in the field, using our proposed dataset: (i) Can model calibration be generalized across different tasks? (ii) Can robustness be used as a calibration measurement? (iii) How reliable are calibration metrics? (iv) Does a post-hoc calibration method affect all models uniformly? (v) How does calibration interact with accuracy? (vi) What is the impact of bin size on calibration measurement? (vii) Which architectural designs are beneficial for calibration? Additionally, our study bridges an existing gap by exploring calibration within NAS. By providing this dataset, we enable further research into NAS calibration. As far as we are aware, our research represents the first large-scale investigation into calibration properties and the premier study of calibration issues within NAS.

深度神经网络在各种机器学习任务中的应用越来越多，但随着这些模型复杂性的增加，尽管预测准确性得到了提高，但它们通常面临校准问题。我们的研究利用神经架构搜索（NAS）搜索空间，为深入探索校准特性提供了一个详尽的模型架构空间，并特别创建一个模型校准数据集，评估了117,702个唯一神经网络在常用的NATS-Bench搜索空间中的90个基于bin的校准度量和12个附加校准度量。同时，本研究探索了NAS中的校准问题，为进一步研究NAS校准提供了可能性。根据我们所了解，我们的研究代表了对校准特性进行的首次大规模调查，也是在NAS中对校准问题进行的首要研究。

关于校准的基准研究