BriefGPT.xyz
Mar, 2020
预训练转换器的校准
Calibration of Pre-trained Transformers
HTML
PDF
Shrey Desai, Greg Durrett
TL;DR
通过对BERT和RoBERTa在自然语言推理、释义检测和常识推理三方面的实验,本研究发现预训练模型在领域内使用时具有校准性,而且与基准相比,在领域外的校准误差可以低至3.5倍;降温和标签平滑等方法可以进一步减少领域内的校准误差和校准领域外的计算后验概率。
Abstract
Pre-trained
transformers
are now ubiquitous in natural language processing, but despite their high end-
task performance
, little is known empirically about whether they are calibrated. Specifically, do these model
→