Achieving precise semantic control over the latent spaces of Variational
AutoEncoders (VAEs) holds significant value for downstream tasks in NLP as the
underlying generative mechanisms could be better localised, explained and
improved upon. Recent research, however, has struggled to achieve consistent
results, primarily due to the inevitable loss of semantic information in the
variational bottleneck and limited control over the decoding mechanism. To
overcome these challenges, we investigate discrete latent spaces in Vector
Quantized Variational AutoEncoders (VQVAEs) to improve semantic control and
generation in Transformer-based VAEs. In particular, We propose T5VQVAE, a
novel model that leverages the controllability of VQVAEs to guide the
self-attention mechanism in T5 at the token-level, exploiting its full
generalization capabilities. Experimental results indicate that T5VQVAE
outperforms existing state-of-the-art VAE models, including Optimus, in terms
of controllability and preservation of semantic information across different
tasks such as auto-encoding of sentences and mathematical expressions, text
transfer, and inference. Moreover, T5VQVAE exhibits improved inference
capabilities, suggesting potential applications for downstream natural language
and symbolic reasoning tasks.

通过使用 Vector Quantized Variational AutoEncoders (VQVAEs) 内的离散潜空间来提高 Transformer-based VAE 中的语义控制和生成性能，我们提出了一种新型模型 T5VQVAE，利用 VQVAEs 的可控性来引导 T5 中的自注意机制，从而更好地保留语义信息并取得优于现有 VAE 模型（包括 Optimus）的性能，在自动编码、文本转换和推理等不同任务中。此外，T5VQVAE 还表现出了改进的推理能力，为下游自然语言和符号推理任务提供了潜在的应用。