A new trend in deep learning, represented by Mutual Information Neural
Estimation (MINE) and Information Noise Contrast Estimation (InfoNCE), is
emerging. In this trend, similarity functions and Estimated Mutual Information
(EMI) are used as learning and objective functions. Coincidentally, EMI is
essentially the same as Semantic Mutual Information (SeMI) proposed by the
author 30 years ago. This paper first reviews the evolutionary histories of
semantic information measures and learning functions. Then, it briefly
introduces the author's semantic information G theory with the rate-fidelity
function R(G) (G denotes SeMI, and R(G) extends R(D)) and its applications to
multi-label learning, the maximum Mutual Information (MI) classification, and
mixture models. Then it discusses how we should understand the relationship
between SeMI and Shan-non's MI, two generalized entropies (fuzzy entropy and
coverage entropy), Autoencoders, Gibbs distributions, and partition functions
from the perspective of the R(G) function or the G theory. An important
conclusion is that mixture models and Restricted Boltzmann Machines converge
because SeMI is maximized, and Shannon's MI is minimized, making information
efficiency G/R close to 1. A potential opportunity is to simplify deep learning
by using Gaussian channel mixture models for pre-training deep neural networks'
latent layers without considering gradients. It also discusses how the SeMI
measure is used as the reward function (reflecting purposiveness) for
reinforcement learning. The G theory helps interpret deep learning but is far
from enough. Combining semantic information theory and deep learning will
accelerate their development.

本文概述了深度学习的新趋势，即利用相似性函数和信息估计函数作为学习和目标函数的相互信息神经估计和信息噪声对比估计，介绍了作者提出的语义信息 G 理论及应用在多标签学习、最大互信息分类和混合模型等方面，探讨了从 G 函数或 G 理论的角度理解语义互信息与 Shanon 互信息、混合模型、Gibbs 分布和分区函数等的关系，并提出了利用高斯信道混合模型来预训练深度神经网络潜在层的潜在机会和使用语义互信息作为增强学习的奖励函数。