Information bottleneck (IB) principle [1] has become an important element in
information-theoretic analysis of deep models. Many state-of-the-art generative
models of both Variational Autoencoder (VAE) [2; 3] and Generative Adversarial
Networks (GAN) [4] families use various bounds on mutual information terms to
introduce certain regularization constraints [5; 6; 7; 8; 9; 10]. Accordingly,
the main difference between these models consists in add regularization
constraints and targeted objectives.
In this work, we will consider the IB framework for three classes of models
that include supervised, unsupervised and adversarial generative models. We
will apply a variational decomposition leading a common structure and allowing
easily establish connections between these models and analyze underlying
assumptions.
Based on these results, we focus our analysis on unsupervised setup and
reconsider the VAE family. In particular, we present a new interpretation of
VAE family based on the IB framework using a direct decomposition of mutual
information terms and show some interesting connections to existing methods
such as VAE [2; 3], beta-VAE [11], AAE [12], InfoVAE [5] and VAE/GAN [13].
Instead of adding regularization constraints to an evidence lower bound (ELBO)
[2; 3], which itself is a lower bound, we show that many known methods can be
considered as a product of variational decomposition of mutual information
terms in the IB framework. The proposed decomposition might also contribute to
the interpretability of generative models of both VAE and GAN families and
create a new insights to a generative compression [14; 15; 16; 17]. It can also
be of interest for the analysis of novelty detection based on one-class
classifiers [18] with the IB based discriminators.

本文基于信息瓶颈原理与变分分解构建了一般的框架结构，分析了现有深度生成模型（VAE，GAN 等）的基本假设与相互之间的联系，提出了新的相互信息分解方式，使得生成模型在压缩生成方面具备新的解释和洞见。