In this paper, we propose Evidential Conformal Prediction (ECP) method for
image classifiers to generate the conformal prediction sets. Our method is
designed based on a non-conformity score function that has its roots in
Evidential Deep Learning (EDL) as a method of quantifying model (epistemic)
uncertainty in DNN classifiers. We use evidence that are derived from the logit
values of target labels to compute the components of our non-conformity score
function: the heuristic notion of uncertainty in CP, uncertainty surprisal, and
expected utility. Our extensive experimental evaluation demonstrates that ECP
outperforms three state-of-the-art methods for generating CP sets, in terms of
their set sizes and adaptivity while maintaining the coverage of true labels.

利用证据一致性预测方法（ECP）为图像分类器生成一致性预测集，通过基于非一致性评分函数，利用目标标签的对数几率值推导出的证据计算非一致性评分函数的组成部分：一致性预测中的不确定性启发式概念、不确定性惊喜度和期望效用，实验评估结果证明，ECP 在生成一致性预测集方面优于三种先进方法，同时保持了对真实标签的覆盖。

基于符合预测的深度分类器的证据不确定性集合

Evidential Uncertainty Sets in Deep Classifiers Using Conformal  Prediction

Current algorithms and architecture can create excellent DNN classifier
models from example data. In general, larger training datasets result in better
model estimations, which improve test performance. Existing methods for
predicting generalization performance are based on hold-out test examples. To
the best of our knowledge, at present no method exists that can estimate the
quality of a trained DNN classifier without test data. In this paper, we show
that the quality of a trained DNN classifier can be assessed without any
example data. We consider DNNs to be composed of a feature extractor and a
feature classifier; the feature extractor's output is fed to the classifier.
The proposed method iteratively creates class prototypes in the input space for
each class by minimizing a cross-entropy loss function at the output of the
network. We use these prototypes and their feature relationships to reveal the
quality of the classifier. We have developed two metrics: one using the
features of the prototypes and the other using adversarial examples
corresponding to each prototype. Empirical evaluations show that accuracy
obtained from test examples is directly proportional to quality measures
obtained from the proposed metrics. We report our observations for ResNet18
with Tiny ImageNet, CIFAR100, and CIFAR10 datasets. The proposed metrics can be
used to compare performances of two or more classifiers without test examples.

该论文提出了一种无需测试数据即可评估训练后 DNN 分类器的质量的方法，通过在输入空间中迭代创建每个类别的类原型来实现，这些原型和其特征关系可用于揭示分类器的质量，并开发了两个使用原型特征和对应于每个原型的对抗性示例的质量度量指标，其准确性与使用测试数据获得的准确性成正比。

如何在没有数据的情况下识别出优秀的深度神经网络分类器

Fantastic DNN Classifiers and How to Identify them without Data

Deep learning systems have been reported to acheive state-of-the-art
performances in many applications, and one of the keys for achieving this is
the existence of well trained classifiers on benchmark datasets which can be
used as backbone feature extractors in downstream tasks. As a main-stream loss
function for training deep neural network (DNN) classifiers, the cross entropy
loss can easily lead us to find models which demonstrate severe overfitting
behavior when no other techniques are used for alleviating it such as data
augmentation. In this paper, we prove that the existing cross entropy loss
minimization for training DNN classifiers essentially learns the conditional
entropy of the underlying data distribution of the dataset, i.e., the
information or uncertainty remained in the labels after revealing the input. In
this paper, we propose a mutual information learning framework where we train
DNN classifiers via learning the mutual information between the label and
input. Theoretically, we give the population error probability lower bound in
terms of the mutual information. In addition, we derive the mutual information
lower and upper bounds for a concrete binary classification data model in
$\mbR^n$, and also the error probability lower bound in this scenario. Besides,
we establish the sample complexity for accurately learning the mutual
information from empirical data samples drawn from the underlying data
distribution. Empirically, we conduct extensive experiments on several
benchmark datasets to support our theory. Without whistles and bells, the
proposed mutual information learned classifiers (MILCs) acheive far better
generalization performances than the state-of-the-art classifiers with an
improvement which can exceed more than 10\% in testing accuracy.

本文用互信息作为代替条件熵来训练深度神经网络分类器，并且在多个基准数据集上进行了实验证明，相对于现有交叉熵损失函数训练的模型，提出的互信息模型 (MILCs) 性能更好，总体测试准确率提升超过 10%。