Recurrent Variational Autoencoder has been widely used for language modeling
and text generation tasks. These models often face a difficult optimization
problem, also known as the Kullback-Leibler (KL) term vanishing issue, where
the posterior easily collapses to the prior, and the model will ignore latent
codes in generative tasks. To address this problem, we introduce an improved
Wasserstein Variational Autoencoder (WAE) with Riemannian Normalizing Flow
(RNF) for text modeling. The RNF transforms a latent variable into a space that
respects the geometric characteristics of input space, which makes posterior
impossible to collapse to the non-informative prior. The Wasserstein objective
minimizes the distance between the marginal distribution and the prior directly
and therefore does not force the posterior to match the prior. Empirical
experiments show that our model avoids KL vanishing over a range of datasets
and has better performances in tasks such as language modeling, likelihood
approximation, and text generation. Through a series of experiments and
analysis over latent space, we show that our model learns latent distributions
that respect latent space geometry and is able to generate sentences that are
more diverse.

通过引入 Riemann 规范流 (RNF) 和 Wasserstein 变分自编码器（WAE），解决了 KL 消失困境，在样本建模、生成任务中的性能更优