Variational autoencoders and Helmholtz machines use a recognition network
(encoder) to approximate the posterior distribution of a generative model
(decoder). In this paper we study the necessary and sufficient properties of a
recognition network so that it can model the true posterior distribution
exactly. These results are derived in the general context of probabilistic
graphical modelling / Bayesian networks, for which the network represents a set
of conditional independence statements. We derive both global conditions, in
terms of d-separation, and local conditions for the recognition network to have
the desired qualities. It turns out that for the local conditions the property
perfectness (for every node, all parents are joined) plays an important role.

本篇论文研究了识别网络模型的 “必要和充分” 属性，使其能够精确地模拟真实的后验分布，结果发现对于局部条件来说， 完美性质 (即对于每个节点，所有的父节点都相连) 发挥了重要作用。

贝叶斯网络的倒置

Inversion of Bayesian Networks

Training deep directed graphical models with many hidden variables and
performing inference remains a major challenge. Helmholtz machines and deep
belief networks are such models, and the wake-sleep algorithm has been proposed
to train them. The wake-sleep algorithm relies on training not just the
directed generative model but also a conditional generative model (the
inference network) that runs backward from visible to latent, estimating the
posterior distribution of latent given visible. We propose a novel
interpretation of the wake-sleep algorithm which suggests that better
estimators of the gradient can be obtained by sampling latent variables
multiple times from the inference network. This view is based on importance
sampling as an estimator of the likelihood, with the approximate inference
network as a proposal distribution. This interpretation is confirmed
experimentally, showing that better likelihood can be achieved with this
reweighted wake-sleep procedure. Based on this interpretation, we propose that
a sigmoidal belief network is not sufficiently powerful for the layers of the
inference network in order to recover a good estimator of the posterior
distribution of latent variables. Our experiments show that using a more
powerful layer model, such as NADE, yields substantially better generative
models.

通过对梯度的多次估计，改进比重加权的 wake-sleep 算法可有效训练 Helmholtz machines 和 deep belief networks 模型，同时 NADE 作为一种更强大的模型代替 sigmoidal belief network 可以更好地估计后验分布。