Mutual information is widely applied to learn latent representations of observations, whilst its implication in classification neural networks remain to be better explained. In this paper, we show that optimising the parameters of classification neural networks with softmax cross-entropy is equivalent to maximising the mutual information between inputs and labels under the balanced data assumption. Through the experiments on synthetic and real datasets, we show that softmax cross-entropy can estimate mutual information approximately. When applied to image classification, this relation helps approximate the point-wise mutual information between an input image and a label without modifying the network structure. In this end, we propose infoCAM, informative class activation map, which highlights regions of the input image that are the most relevant to a given label based on differences in information. The activation map helps localise the target object in an image. Through the experiments on the semi-supervised object localisation task with two real-world datasets, we evaluate the effectiveness of the information-theoretic approach.

本文提出了一种信息论方法：通过用 softmax 交叉熵优化分类神经网络的参数，来最大化假设数据平衡下输入和标签之间的互信息，并通过在基于差异信息的 informativeness class activation map 中突出显示输入图像中与给定标签相关性最高的区域来帮助定位目标对象，并通过使用两个真实世界数据集上半监督目标定位任务的实验来评估我们信息理论方法的有效性。

重新思考Softmax：神经网络分类器作为互信息估计器的角色