Several recent works find empirically that the average test error of deep
neural networks can be estimated via the prediction disagreement of models,
which does not require labels. In particular, Jiang et al. (2022) show for the
disagreement between two separately trained networks that this `Generalization
Disagreement Equality' follows from the well-calibrated nature of deep
ensembles under the notion of a proposed `class-aggregated calibration.' In
this reproduction, we show that the suggested theory might be impractical
because a deep ensemble's calibration can deteriorate as prediction
disagreement increases, which is precisely when the coupling of test error and
disagreement is of interest, while labels are needed to estimate the
calibration on new datasets. Further, we simplify the theoretical statements
and proofs, showing them to be straightforward within a probabilistic context,
unlike the original hypothesis space view employed by Jiang et al. (2022).

本文介绍了深度神经网络的预测误差可以通过模型的预测不一致性进行估计，其中，“广义差异等式” 遵循了深度集合的良好校准性，并提出了 “类别聚合校准” 的概念。但我们发现该理论可能不实用，因为预测不一致性增加时，深度集合的校准性可能会降低，并需要标签来估计新数据集上的校准性。我们简化了理论陈述和证明，并在概率上下文中展示它们是简单明了的。

关于 “通过差异性评估 SGD 的泛化能力” 注释

A Note on "Assessing Generalization of SGD via Disagreement"

Recent work has shown that models trained to the same objective, and which
achieve similar measures of accuracy on consistent test data, may nonetheless
behave very differently on individual predictions. This inconsistency is
undesirable in high-stakes contexts, such as medical diagnosis and finance. We
show that this inconsistent behavior extends beyond predictions to feature
attributions, which may likewise have negative implications for the
intelligibility of a model, and one's ability to find recourse for subjects. We
then introduce selective ensembles to mitigate such inconsistencies by applying
hypothesis testing to the predictions of a set of models trained using
randomly-selected starting conditions; importantly, selective ensembles can
abstain in cases where a consistent outcome cannot be achieved up to a
specified confidence level. We prove that that prediction disagreement between
selective ensembles is bounded, and empirically demonstrate that selective
ensembles achieve consistent predictions and feature attributions while
maintaining low abstention rates. On several benchmark datasets, selective
ensembles reach zero inconsistently predicted points, with abstention rates as
low 1.5%.

通过引入选择性集合来减轻模型的不一致性，使用随机选定的起始条件训练一组模型的预测上执行假设检验；在这些情况下，选择性集合可以放弃不一致预测结果并在指定的置信水平下达成一致结果，多个基准数据集上展示出零一致性预测点和低至 1.5％的放弃率。