While most machine learning models can provide confidence in their predictions, confidence is insufficient to understand a prediction's reliability. For instance, the model may have a low confidence prediction if the input is not well-represented in the training dataset or if the input is inherently ambiguous. In this work, we investigate the relationship between how atypical(rare) a sample or a class is and the reliability of a model's predictions. We first demonstrate that atypicality is strongly related to miscalibration and accuracy. In particular, we empirically show that predictions for atypical inputs or atypical classes are more overconfident and have lower accuracy. Using these insights, we show incorporating atypicality improves uncertainty quantification and model performance for discriminative neural networks and large language models. In a case study, we show that using atypicality improves the performance of a skin lesion classifier across different skin tone groups without having access to the group attributes. Overall, we propose that models should use not only confidence but also atypicality to improve uncertainty quantification and performance. Our results demonstrate that simple post-hoc atypicality estimators can provide significant value.

本论文研究了机器学习模型的可靠性问题，发现输入信息的典型性与模型预测的准确性和过度自信程度有关，提出使用输入信息的典型性来改进模型的性能和不确定性估计。通过案例研究，展示了该方法可以在不获取群组属性的情况下提高人类皮肤病变分类器在不同肤色群体中的性能。

超越置信度：可靠的模型应该考虑到非典型性