We introduce the Mutual Information Machine (MIM), a probabilistic
auto-encoder for learning joint distributions over observations and latent
variables. MIM reflects three design principles: 1) low divergence, to
encourage the encoder and decoder to learn consistent factorizations of the
same underlying distribution; 2) high mutual information, to encourage an
informative relation between data and latent variables; and 3) low marginal
entropy, or compression, which tends to encourage clustered latent
representations. We show that a combination of the Jensen-Shannon divergence
and the joint entropy of the encoding and decoding distributions satisfies
these criteria, and admits a tractable cross-entropy bound that can be
optimized directly with Monte Carlo and stochastic gradient descent. We
contrast MIM learning with maximum likelihood and VAEs. Experiments show that
MIM learns representations with high mutual information, consistent encoding
and decoding distributions, effective latent clustering, and data log
likelihood comparable to VAE, while avoiding posterior collapse.

引入 Mutual Information Machine (MIM)，一种概率自编码器，用于学习观测和潜变量的联合分布。MIM 反映了三个设计原则：低散度、高互信息和低边际熵，这有助于学习聚类潜在表示。与极大似然法和变分自编码器 (VAE) 相比，实验表明，MIM 通过避免后验崩溃，学习具有高互信息和一致的编码和解码分布的表征。