In this note we present a generative model of natural images consisting of a
deep hierarchy of layers of latent random variables, each of which follows a
new type of distribution that we call rectified Gaussian. These rectified
Gaussian units allow spike-and-slab type sparsity, while retaining the
differentiability necessary for efficient stochastic gradient variational
inference. To learn the parameters of the new model, we approximate the
posterior of the latent variables with a variational auto-encoder. Rather than
making the usual mean-field assumption however, the encoder parameterizes a new
type of structured variational approximation that retains the prior
dependencies of the generative model. Using this structured posterior
approximation, we are able to perform joint training of deep models with many
layers of latent random variables, without having to resort to stacking or
other layerwise training procedures.

本研究提出了一种基于深度潜在随机变量的自然图像生成模型，其采用新型分布称为修正高斯，其中采用类似 spike-and-slab 的稀疏性，保持了有效的随机梯度变分推断的可微性；通过一个结构化后验分布估计函数的近似，提出了一种新型结构化变分近似方法，避免常规均场假设，并保持了生成模型的先验依赖关系，从而实现了具有许多层潜在随机变量的深度模型的联合训练。