Pre-trained language models (PLMs) have gained increasing popularity due to
their compelling prediction performance in diverse natural language processing
(NLP) tasks. When formulating a PLM-based prediction pipeline for NLP tasks, it
is also crucial for the pipeline to minimize the calibration error, especially
in safety-critical applications. That is, the pipeline should reliably indicate
when we can trust its predictions. In particular, there are various
considerations behind the pipeline: (1) the choice and (2) the size of PLM, (3)
the choice of uncertainty quantifier, (4) the choice of fine-tuning loss, and
many more. Although prior work has looked into some of these considerations,
they usually draw conclusions based on a limited scope of empirical studies.
There still lacks a holistic analysis on how to compose a well-calibrated
PLM-based prediction pipeline. To fill this void, we compare a wide range of
popular options for each consideration based on three prevalent NLP
classification tasks and the setting of domain shift. In response, we recommend
the following: (1) use ELECTRA for PLM encoding, (2) use larger PLMs if
possible, (3) use Temp Scaling as the uncertainty quantifier, and (4) use Focal
Loss for fine-tuning.

该论文研究了基于预训练语言模型的 NLP 任务中的预测流水线如何最小化校准误差，比较了不同选项的表现，并推荐使用 ELECTRA 作为 PLM 编码，尽可能使用更大的 PLM，使用温度调节作为不确定性度量，以及使用 Focal Loss 进行微调。