Applying a machine learning model for decision-making in the real world
requires to distinguish what the model knows from what it does not. A critical
factor in assessing the knowledge of a model is to quantify its predictive
uncertainty. Predictive uncertainty is commonly measured by the entropy of the
Bayesian model average (BMA) predictive distribution. Yet, the properness of
this current measure of predictive uncertainty was recently questioned. We
provide new insights regarding those limitations. Our analyses show that the
current measure erroneously assumes that the BMA predictive distribution is
equivalent to the predictive distribution of the true model that generated the
dataset. Consequently, we introduce a theoretically grounded measure to
overcome these limitations. We experimentally verify the benefits of our
introduced measure of predictive uncertainty. We find that our introduced
measure behaves more reasonably in controlled synthetic tasks. Moreover, our
evaluations on ImageNet demonstrate that our introduced measure is advantageous
in real-world applications utilizing predictive uncertainty.

应用机器学习模型进行决策需要区分模型所知与不知的内容。本文提供新的洞见，揭示当前预测不确定性度量的局限性，并引入了一种理论上可靠的度量方法来克服这些局限。实验证明，我们引入的度量方法在控制的合成任务中表现得更加合理，并且在利用预测不确定性的实际应用中具有优势。