Variational autoencoders (VAEs) are one of the powerful unsupervised learning
frameworks in NLP for latent representation learning and latent-directed
generation. The classic optimization goal of VAEs is to maximize the Evidence
Lower Bound (ELBo), which consists of a conditional likelihood for generation
and a negative Kullback-Leibler (KL) divergence for regularization. In
practice, optimizing ELBo often leads the posterior distribution of all samples
converge to the same degenerated local optimum, namely posterior collapse or KL
vanishing. There are effective ways proposed to prevent posterior collapse in
VAEs, but we observe that they in essence make trade-offs between posterior
collapse and hole problem, i.e., mismatch between the aggregated posterior
distribution and the prior distribution. To this end, we introduce new training
objectives to tackle both two problems through a novel regularization based on
the probabilistic density gap between the aggregated posterior distribution and
the prior distribution. Through experiments on language modeling, latent space
visualization and interpolation, we show that our proposed method can solve
both problems effectively and thus outperforms the existing methods in
latent-directed generation. To the best of our knowledge, we are the first to
jointly solve the hole problem and the posterior collapse.

通过在概率密度差异方面引入新的正则化方法，有效解决了 Variational autoencoders 中的 LATENT REPRESENTATION LEARNING 方面出现的后验崩溃和空洞问题。