Variational dimensionality reduction methods are known for their high
accuracy, generative abilities, and robustness. These methods have many
theoretical justifications. Here we introduce a unifying principle rooted in
information theory to rederive and generalize existing variational methods and
design new ones. We base our framework on an interpretation of the multivariate
information bottleneck, in which two Bayesian networks are traded off against
one another. We interpret the first network as an encoder graph, which
specifies what information to keep when compressing the data. We interpret the
second network as a decoder graph, which specifies a generative model for the
data. Using this framework, we rederive existing dimensionality reduction
methods such as the deep variational information bottleneck (DVIB), beta
variational auto-encoders (beta-VAE), and deep variational canonical
correlation analysis (DVCCA). The framework naturally introduces a trade-off
parameter between compression and reconstruction in the DVCCA family of
algorithms, resulting in the new beta-DVCCA family. In addition, we derive a
new variational dimensionality reduction method, deep variational symmetric
informational bottleneck (DVSIB), which simultaneously compresses two variables
to preserve information between their compressed representations. We implement
all of these algorithms and evaluate their ability to produce shared low
dimensional latent spaces on a modified noisy MNIST dataset. We show that
algorithms that are better matched to the structure of the data (beta-DVCCA and
DVSIB) produce better latent spaces as measured by classification accuracy and
the dimensionality of the latent variables. We believe that this framework can
be used to unify other multi-view representation learning algorithms.
Additionally, it provides a straightforward framework for deriving
problem-specific loss functions.

通过信息论，我们提出了一个可以重新演绎和推广现有变分方法，并设计新方法的统一原则。我们的框架基于多变量信息瓶颈的解释，其中两个贝叶斯网络相互权衡。我们将第一个网络解释为编码器图，指定了在压缩数据时要保留哪些信息；我们将第二个网络解释为解码器图，为数据指定了生成模型。利用这个框架，我们重新演绎了现有的降维方法，如深度变分信息瓶颈 (DVIB)，Beta 变分自动编码器 (beta-VAE) 和深度变分规范相关分析 (DVCCA)。该框架在 DVCCA 算法族中自然引入了一个压缩和重建之间的权衡参数，从而产生了新的 Beta-DVCCA 算法族。此外，我们推导出了一种新的变分降维方法，深度变分对称信息瓶颈 (DVSIB)，它可以同时压缩两个变量以保留它们压缩表示之间的信息。我们实现了所有这些算法，并在经过修改的噪声 MNIST 数据集上评估它们产生共享低维潜空间的能力。我们展示了与数据结构更匹配的算法 (Beta-DVCCA 和 DVSIB) 如何通过分类准确性和潜变量的维度来测量产生更好的潜空间。我们相信这个框架可以用来统一其他多视图表示学习算法，此外，它还为推导问题特定的损失函数提供了一个直观的框架。