Automatic Differentiation Variational Inference (ADVI) is a useful tool for
efficiently learning probabilistic models in machine learning. Generally
approximate posteriors learned by ADVI are forced to be unimodal in order to
facilitate use of the reparameterization trick. In this paper, we show how
stratified sampling may be used to enable mixture distributions as the
approximate posterior, and derive a new lower bound on the evidence analogous
to the importance weighted autoencoder (IWAE). We show that this "SIWAE" is a
tighter bound than both IWAE and the traditional ELBO, both of which are
special instances of this bound. We verify empirically that the traditional
ELBO objective disfavors the presence of multimodal posterior distributions and
may therefore not be able to fully capture structure in the latent space. Our
experiments show that using the SIWAE objective allows the encoder to learn
more complex distributions which regularly contain multimodality, resulting in
higher accuracy and better calibration in the presence of incomplete, limited,
or corrupted data.

本论文引入了分层抽样的方法，实现了对混合分布的自动微分变分推断（ADVI）； 并且通过建立重要性加权自编码器（IWAE）类似的较紧下界，进一步验证了该方法相较于传统 ELBO 或 IWAE 的优势，特别是在 latents 变量中有多峰分布时，表现出更高的准确性和更好的校准性。

混合自动微分变分推断

Automatic Differentiation Variational Inference with Mixtures

The variational autoencoder (VAE; Kingma, Welling (2014)) is a recently
proposed generative model pairing a top-down generative network with a
bottom-up recognition network which approximates posterior inference. It
typically makes strong assumptions about posterior inference, for instance that
the posterior distribution is approximately factorial, and that its parameters
can be approximated with nonlinear regression from the observations. As we show
empirically, the VAE objective can lead to overly simplified representations
which fail to use the network's entire modeling capacity. We present the
importance weighted autoencoder (IWAE), a generative model with the same
architecture as the VAE, but which uses a strictly tighter log-likelihood lower
bound derived from importance weighting. In the IWAE, the recognition network
uses multiple samples to approximate the posterior, giving it increased
flexibility to model complex posteriors which do not fit the VAE modeling
assumptions. We show empirically that IWAEs learn richer latent space
representations than VAEs, leading to improved test log-likelihood on density
estimation benchmarks.

本文提出了 IWAE，一种基于 VAE 的生成模型，将后验分布建模的灵活性提高，使得学习到的潜空间表示更加丰富，从而取得更好的密度估计结果。