Although sparse training has been successfully used in various
resource-limited deep learning tasks to save memory, accelerate training, and
reduce inference time, the reliability of the produced sparse models remains
unexplored. Previous research has shown that deep neural networks tend to be
over-confident, and we find that sparse training exacerbates this problem.
Therefore, calibrating the sparse models is crucial for reliable prediction and
decision-making. In this paper, we propose a new sparse training method to
produce sparse models with improved confidence calibration. In contrast to
previous research that uses only one mask to control the sparse topology, our
method utilizes two masks, including a deterministic mask and a random mask.
The former efficiently searches and activates important weights by exploiting
the magnitude of weights and gradients. While the latter brings better
exploration and finds more appropriate weight values by random updates.
Theoretically, we prove our method can be viewed as a hierarchical variational
approximation of a probabilistic deep Gaussian process. Extensive experiments
on multiple datasets, model architectures, and sparsities show that our method
reduces ECE values by up to 47.8\% and simultaneously maintains or even
improves accuracy with only a slight increase in computation and storage
burden.

本研究提出了一种新的稀疏训练方法，利用两个掩码来生产具备优化置信度校准的稀疏模型，文中实验证明，该方法使 ECE 值下降了达 47.8％，并且同时保持或提高准确性，而只需稍微增加计算和存储负担。