Mutual Information (MI) has been widely used as a loss regularizer for
training neural networks. This has been particularly effective when learn
disentangled or compressed representations of high dimensional data. However,
differential entropy (DE), another fundamental measure of information, has not
found widespread use in neural network training. Although DE offers a
potentially wider range of applications than MI, off-the-shelf DE estimators
are either non differentiable, computationally intractable or fail to adapt to
changes in the underlying distribution. These drawbacks prevent them from being
used as regularizers in neural networks training. To address shortcomings in
previously proposed estimators for DE, here we introduce KNIFE, a fully
parameterized, differentiable kernel-based estimator of DE. The flexibility of
our approach also allows us to construct KNIFE-based estimators for conditional
(on either discrete or continuous variables) DE, as well as MI. We empirically
validate our method on high-dimensional synthetic data and further apply it to
guide the training of neural networks for real-world tasks. Our experiments on
a large variety of tasks, including visual domain adaptation, textual fair
classification, and textual fine-tuning demonstrate the effectiveness of
KNIFE-based estimation. Code can be found at
this https URL

介绍了一种全参数化、可微分的基于核的差分熵估计器 KNIFE，可以在神经网络训练中作为正则化器，通过大量实验证明了 KNIFE 的有效性。