Large-scale pre-trained models have achieved remarkable success in a variety
of scenarios and applications, but how to leverage them to improve the
prediction reliability of downstream models is undesirably under-explored.
Moreover, modern neural networks have been found to be poorly calibrated and
make overconfident predictions regardless of inherent sample difficulty and
data uncertainty. To address this issue, we propose to utilize large-scale
pre-trained models to guide downstream model training with sample
difficulty-aware entropy regularization. Pre-trained models that have been
exposed to large-scale datasets and do not overfit the downstream training
classes enable us to measure each training sample difficulty via feature-space
Gaussian modeling and relative Mahalanobis distance computation. Importantly,
by adaptively penalizing overconfident prediction based on the sample's
difficulty, we simultaneously improve accuracy and uncertainty calibration on
various challenging benchmarks, consistently surpassing competitive baselines
for reliable prediction.

本文提出了一个方法，即基于样本难度感知熵正则化，利用经过大规模训练的预先训练模型指导下游模型训练，并同时提高了准确性和不确定性校准的可靠预测，克服了现代神经网络的困难，这是一个未被充分探索的领域。