Despite the significant progress made in practical applications of aligned
language models (LMs), they tend to be overconfident in output answers compared
to the corresponding pre-trained LMs. In this work, we systematically evaluate
the impact of the alignment process on logit-based uncertainty calibration of
LMs under the multiple-choice setting. We first conduct a thoughtful empirical
study on how aligned LMs differ in calibration from their pre-trained
counterparts. Experimental results reveal that there are two distinct
uncertainties in LMs under the multiple-choice setting, which are responsible
for the answer decision and the format preference of the LMs, respectively.
Then, we investigate the role of these two uncertainties on aligned LM's
calibration through fine-tuning in simple synthetic alignment schemes and
conclude that one reason for aligned LMs' overconfidence is the conflation of
these two types of uncertainty. Furthermore, we examine the utility of common
post-hoc calibration methods for aligned LMs and propose an easy-to-implement
and sample-efficient method to calibrate aligned LMs. We hope our findings
could provide insights into the design of more reliable alignment processes for
LMs.

对齐的语言模型在多选题设置下多显示过于自信的输出答案，我们系统评估了对齐过程对语言模型的逻辑回归置信度校准的影响，并提出了易于实施且高效的校准方法。

研究多选设置下对齐语言模型的不确定性校准

Investigating Uncertainty Calibration of Aligned Language Models under  the Multiple-Choice Setting

The class-wise training losses often diverge as a result of the various
levels of intra-class and inter-class appearance variation, and we find that
the diverging class-wise training losses cause the uncalibrated prediction with
its reliability. To resolve the issue, we propose a new calibration method to
synchronize the class-wise training losses. We design a new training loss to
alleviate the variance of class-wise training losses by using multiple
class-wise scaling factors. Since our framework can compensate the training
losses of overfitted classes with those of under-fitted classes, the integrated
training loss is preserved, preventing the performance drop even after the
model calibration. Furthermore, our method can be easily employed in the
post-hoc calibration methods, allowing us to use the pre-trained model as an
initial model and reduce the additional computation for model calibration. We
validate the proposed framework by employing it in the various post-hoc
calibration methods, which generally improves calibration performance while
preserving accuracy, and discover through the investigation that our approach
performs well with unbalanced datasets and untuned hyperparameters.

为解决各个类别间外观变化的不同导致的严重训练偏差和预测不可靠性的问题，本研究提出了一种新的预测校准方法，通过使用多个类别缩放因子缓解类别训练误差的差异。实验表明该方法在类别不平衡和超参数未经调整的情况下表现出色，可轻松与后续校准方法结合使用。