Deep ensembles perform better than a single network thanks to the diversity
among their members. Recent approaches regularize predictions to increase
diversity; however, they also drastically decrease individual members'
performances. In this paper, we argue that learning strategies for deep
ensembles need to tackle the trade-off between ensemble diversity and
individual accuracies. Motivated by arguments from information theory and
leveraging recent advances in neural estimation of conditional mutual
information, we introduce a novel training criterion called DICE: it increases
diversity by reducing spurious correlations among features. The main idea is
that features extracted from pairs of members should only share information
useful for target class prediction without being conditionally redundant.
Therefore, besides the classification loss with information bottleneck, we
adversarially prevent features from being conditionally predictable from each
other. We manage to reduce simultaneous errors while protecting class
information. We obtain state-of-the-art accuracy results on CIFAR-10/100: for
example, an ensemble of 5 networks trained with DICE matches an ensemble of 7
networks trained independently. We further analyze the consequences on
calibration, uncertainty estimation, out-of-distribution detection and online
co-distillation.

使用新型训练准则 DICE，提高深度集成模型的多样性并同时保持准确性，可以在 CIFAR-10 和 CIFAR-100 数据集上获得最新的最高准确率结果，并影响了校准、不确定性估计和在线协同训练。

DICE: 基于条件冗余对抗估计的深度集成多样性

DICE: Diversity in Deep Ensembles via Conditional Redundancy Adversarial  Estimation

Since its inception, the neural estimation of mutual information (MI) has
demonstrated the empirical success of modeling expected dependency between
high-dimensional random variables. However, MI is an aggregate statistic and
cannot be used to measure point-wise dependency between different events. In
this work, instead of estimating the expected dependency, we focus on
estimating point-wise dependency (PD), which quantitatively measures how likely
two outcomes co-occur. We show that we can naturally obtain PD when we are
optimizing MI neural variational bounds. However, optimizing these bounds is
challenging due to its large variance in practice. To address this issue, we
develop two methods (free of optimizing MI variational bounds): Probabilistic
Classifier and Density-Ratio Fitting. We demonstrate the effectiveness of our
approaches in 1) MI estimation, 2) self-supervised representation learning, and
3) cross-modal retrieval task.

本文提出了两种方法（Probabilistic Classifier 和 Density-Ratio Fitting）来解决 MI 估计中存在的方差大的问题，这两种方法可以自然地获得点对点的依赖信息，为自监督学习和跨模态检索任务提供了有效方法。

基于神经网络的逐点依存关系估计方法

Neural Methods for Point-wise Dependency Estimation

We point out a limitation of the mutual information neural estimation (MINE)
where the network fails to learn at the initial training phase, leading to slow
convergence in the number of training iterations. To solve this problem, we
propose a faster method called the mutual information neural entropic
estimation (MI-NEE). Our solution first generalizes MINE to estimate the
entropy using a custom reference distribution. The entropy estimate can then be
used to estimate the mutual information. We argue that the seemingly redundant
intermediate step of entropy estimation allows one to improve the convergence
by an appropriate reference distribution. In particular, we show that MI-NEE
reduces to MINE in the special case when the reference distribution is the
product of marginal distributions, but faster convergence is possible by
choosing the uniform distribution as the reference distribution instead.
Compared to the product of marginals, the uniform distribution introduces more
samples in low-density regions and fewer samples in high-density regions, which
appear to lead to an overall larger gradient for faster convergence.

提出了一种更快的方法，称为互信息神经熵估计（MI-NEE），其将互信息估计变为估计熵的过程，通过使用自定义参考分布和选择均匀分布作为参考分布来实现更快的收敛。