Variational Auto-Encoder (VAE) has been widely adopted in text generation. Among many variants, recurrent VAE learns token-wise latent variables with each conditioned on the preceding ones, which captures sequential variability better in the era of RNN. However, it is unclear how to incorporate such recurrent dynamics into the recently dominant Transformer due to its parallelism. In this work, we propose DELLA, a Transformer-based recurrent VAE structure. DELLA imposes recurrence on segment-wise latent variables with arbitrarily separated text segments and constructs the posterior distribution with residual parameterization. Besides, we design an acceleration method by approximating idempotent matrices, which allows parallelism while maintaining the conditional dependence of latent variables. We demonstrate that DELLA could enhance the entanglement of each segment and preceding latent variables and deduce a non-zero lower bound of the KL term, providing a theoretical guarantee of generation diversity. Experiments on two unconditional and one conditional generation tasks show that DELLA achieves significantly improved diversity while maintaining satisfactory generation quality.

本文提出了TRACE，一种用Transformer作为基础的循环VAE结构，可以增强文本段之间的交错关系，提高生成多样性，并在不降低生成质量的情况下取得明显改进。

循环增强多样性！重新审视变分自编码器中的循环潜变量用于产生多样文本