Large pre-trained language models (PLMs) have demonstrated strong performance
on natural language understanding (NLU) tasks through fine-tuning. However,
fine-tuned models still suffer from overconfident predictions, especially in
out-of-domain settings. In this paper, we tackle the problem of calibrating
fine-tuned language models. We demonstrate that the PLMs are well-calibrated on
the masked language modeling task with robust predictive confidence under
domain shift, yet the fine-tuned models fail to retain such property due to
catastrophic forgetting, which impacts the calibration on the downstream
classification task. In light of these observations, we evaluate the
calibration of several methods that preserve pre-trained features and show that
preserving pre-trained features can improve the calibration of fine-tuned
language models. Among these methods, our proposed method that encourages the
fine-tuned model to learn generative representations with auxiliary language
modeling objective achieves competitive accuracy and the lowest expected
calibration error compared to several strong baselines under both in-domain and
out-of-domain settings on three downstream NLU tasks.

本研究主要探讨了针对领域偏移的细调语言模型进行校准的问题，并提出了一种有效的方法，即在辅助语言建模目标的基础上鼓励细调模型学习生成表示，从而提高其校准度并在三项下游自然语言理解任务中具有良好的表现。