Recently, several methods have been proposed for estimating the mutual
information from sample data using deep neural networks and without the knowing
closed form distribution of the data. This class of estimators is referred to
as neural mutual information estimators. Although very promising, such
techniques have yet to be rigorously bench-marked so as to establish their
efficacy, ease of implementation, and stability for capacity estimation which
is joint maximization frame-work. In this paper, we compare the different
techniques proposed in the literature for estimating capacity and provide a
practitioner perspective on their effectiveness. In particular, we study the
performance of mutual information neural estimator (MINE), smoothed mutual
information lower-bound estimator (SMILE), and directed information neural
estimator (DINE) and provide insights on InfoNCE. We evaluated these algorithms
in terms of their ability to learn the input distributions that are capacity
approaching for the AWGN channel, the optical intensity channel, and peak
power-constrained AWGN channel. For both scenarios, we provide insightful
comments on various aspects of the training process, such as stability,
sensitivity to initialization.

本研究对目前文献提出的神经互信息估计器如 MINE、SMILE、DINE 进行了对比评估，以探讨它们在容量估计方面的表现和应用效果。

神经容量估计器：它们的可靠性如何？

Neural Capacity Estimators: How Reliable Are They?

Measuring Mutual Information (MI) between high-dimensional, continuous,
random variables from observed samples has wide theoretical and practical
applications. Recent work, MINE (Belghazi et al. 2018), focused on estimating
tight variational lower bounds of MI using neural networks, but assumed
unlimited supply of samples to prevent overfitting. In real world applications,
data is not always available at a surplus. In this work, we focus on improving
data efficiency and propose a Data-Efficient MINE Estimator (DEMINE), by
developing a relaxed predictive MI lower bound that can be estimated at higher
data efficiency by orders of magnitudes. The predictive MI lower bound also
enables us to develop a new meta-learning approach using task augmentation,
Meta-DEMINE, to improve generalization of the network and further boost
estimation accuracy empirically. With improved data-efficiency, our estimators
enables statistical testing of dependency at practical dataset sizes. We
demonstrate the effectiveness of our estimators on synthetic benchmarks and a
real world fMRI data, with application of inter-subject correlation analysis.

本文提出了一种基于神经网络的数据效率更高的 Mutual Information 估计方法 DEMINE，通过放松预测 MI 下限来提高数据效率，并采用任务扩充方法 Meta-DEMINE 进一步优化其推广性和估计准确性，可用于实际数据集大小的统计依赖性检测。

数据高效的互信息神经估计器

Data-Efficient Mutual Information Neural Estimator

We argue that the estimation of mutual information between high dimensional
continuous random variables can be achieved by gradient descent over neural
networks. We present a Mutual Information Neural Estimator (MINE) that is
linearly scalable in dimensionality as well as in sample size, trainable
through back-prop, and strongly consistent. We present a handful of
applications on which MINE can be used to minimize or maximize mutual
information. We apply MINE to improve adversarially trained generative models.
We also use MINE to implement Information Bottleneck, applying it to supervised
classification; our results demonstrate substantial improvement in flexibility
and performance in these settings.

本文提出了互信息神经估计（MINE），利用神经网络通过梯度下降来估计高维连续随机变量之间的互信息，并使用 MINE 来改进对抗训练的生成模型，并应用信息瓶颈来进行监督分类，实现了灵活性和性能的显著提高。