This paper studies the fundamental problem of learning multi-layer generator models. The multi-layer generator model builds multiple layers of latent variables as a prior model on top of the generator, which benefits learning complex data distribution and hierarchical representations. However, such a prior model usually focuses on modeling inter-layer relations between latent variables by assuming non-informative (conditional) Gaussian distributions, which can be limited in model expressivity. To tackle this issue and learn more expressive prior models, we propose an energy-based model (EBM) on the joint latent space over all layers of latent variables with the multi-layer generator as its backbone. Such joint latent space EBM prior model captures the intra-layer contextual relations at each layer through layer-wise energy terms, and latent variables across different layers are jointly corrected. We develop a joint training scheme via maximum likelihood estimation (MLE), which involves Markov Chain Monte Carlo (MCMC) sampling for both prior and posterior distributions of the latent variables from different layers. To ensure efficient inference and learning, we further propose a variational training scheme where an inference model is used to amortize the costly posterior MCMC sampling. Our experiments demonstrate that the learned model can be expressive in generating high-quality images and capturing hierarchical features for better outlier detection.

本文提出了一种基于能量的模型(EBM)方法，通过在所有层的潜在变量上建立联合潜在空间EBM先验模型，以层内能量项捕捉每一层的内部上下文关系，跨不同层的潜在变量则通过联合校正。并提出联合训练方案(最大似然估计MLE和基于推理模型的变分训练方案)，使得学到的模型可以在生成高质量图像和捕捉分层特征方面具有表现力和更好的异常检测效果。

多层生成器学习联合潜空间EBM先验模型