Estimating the uncertainty of a model's prediction on a test point is a
crucial part of ensuring reliability and calibration under distribution shifts.
A minimum description length approach to this problem uses the predictive
normalized maximum likelihood (pNML) distribution, which considers every
possible label for a data point, and decreases confidence in a prediction if
other labels are also consistent with the model and training data. In this work
we propose IF-COMP, a scalable and efficient approximation of the pNML
distribution that linearizes the model with a temperature-scaled Boltzmann
influence function. IF-COMP can be used to produce well-calibrated predictions
on test points as well as measure complexity in both labelled and unlabelled
settings. We experimentally validate IF-COMP on uncertainty calibration,
mislabel detection, and OOD detection tasks, where it consistently matches or
beats strong baseline methods.

用 IF-COMP 方法，通过温度缩放的玻尔兹曼影响函数来线性化模型，从而实现对测试点的可靠性校准和复杂度测量，提供准确的预测、复杂度和误标记检测等功能。

用 Boltzmann 影响函数衡量随机数据复杂性

Measuring Stochastic Data Complexity with Boltzmann Influence Functions

The Predictive Normalized Maximum Likelihood (pNML) scheme has been recently
suggested for universal learning in the individual setting, where both the
training and test samples are individual data. The goal of universal learning
is to compete with a ``genie'' or reference learner that knows the data values,
but is restricted to use a learner from a given model class. The pNML minimizes
the associated regret for any possible value of the unknown label. Furthermore,
its min-max regret can serve as a pointwise measure of learnability for the
specific training and data sample. In this work we examine the pNML and its
associated learnability measure for the Deep Neural Network (DNN) model class.
As shown, the pNML outperforms the commonly used Empirical Risk Minimization
(ERM) approach and provides robustness against adversarial attacks. Together
with its learnability measure it can detect out of distribution test examples,
be tolerant to noisy labels and serve as a confidence measure for the ERM.
Finally, we extend the pNML to a ``twice universal'' solution, that provides
universality for model class selection and generates a learner competing with
the best one from all model classes.

研究了深度神经网络模型类中预测归一化极大似然（pNML）方案及其相关可学习性度量，发现在对抗攻击方面具有鲁棒性，可用于检测样本类别分布不均、噪声标签等情况，且可作为基准学习者竞争的学习器生成 “两次通用” 的解决方案。

深度 pNML：深度神经网络的预测性归一化最大似然

Deep pNML: Predictive Normalized Maximum Likelihood for Deep Neural  Networks

Universal supervised learning is considered from an information theoretic
point of view following the universal prediction approach, see Merhav and Feder
(1998). We consider the standard supervised "batch" learning where prediction
is done on a test sample once the entire training data is observed, and the
individual setting where the features and labels, both in the training and
test, are specific individual quantities. The information theoretic approach
naturally uses the self-information loss or log-loss. Our results provide
universal learning schemes that compete with a "genie" (or reference) that
knows the true test label. In particular, it is demonstrated that the main
proposed scheme, termed Predictive Normalized Maximum Likelihood (pNML), is a
robust learning solution that outperforms the current leading approach based on
Empirical Risk Minimization (ERM). Furthermore, the pNML construction provides
a pointwise indication for the learnability of the specific test challenge with
the given training examples

本文从信息论的角度探讨了通用监督学习问题，提出了一个名为 Predictive Normalized Maximum Likelihood (pNML) 的学习方案，并证明其在特定测试任务的学习能力超过了当前基于经验风险最小化 (ERM) 的领先方法。