Variational mutual information (MI) estimators are widely used in
unsupervised representation learning methods such as contrastive predictive
coding (CPC). A lower bound on MI can be obtained from a multi-class
classification problem, where a critic attempts to distinguish a positive
sample drawn from the underlying joint distribution from $(m-1)$ negative
samples drawn from a suitable proposal distribution. Using this approach, MI
estimates are bounded above by $\log m$, and could thus severely underestimate
unless $m$ is very large. To overcome this limitation, we introduce a novel
estimator based on a multi-label classification problem, where the critic needs
to jointly identify multiple positive samples at the same time. We show that
using the same amount of negative samples, multi-label CPC is able to exceed
the $\log m$ bound, while still being a valid lower bound of mutual
information. We demonstrate that the proposed approach is able to lead to
better mutual information estimation, gain empirical improvements in
unsupervised representation learning, and beat a current state-of-the-art
knowledge distillation method over 10 out of 13 tasks.

本文提出了一种基于多标签分类问题的新的变分互信息估计器，能够超越对数上界且仍然是互信息的一个有效下界，该方法具有更好的互信息估计性能，在无监督表示学习中实现了实证改进，并在 13 项任务中超越了当前最先进的知识蒸馏方法。