The injection of syntactic information in Variational AutoEncoders (VAEs) has
been shown to result in an overall improvement of performances and
generalisation. An effective strategy to achieve such a goal is to separate the
encoding of distributional semantic features and syntactic structures into
heterogeneous latent spaces via multi-task learning or dual encoder
architectures. However, existing works employing such techniques are limited to
LSTM-based VAEs. In this paper, we investigate latent space separation methods
for structural syntactic injection in Transformer-based VAE architectures
(i.e., Optimus). Specifically, we explore how syntactic structures can be
leveraged in the encoding stage through the integration of graph-based and
sequential models, and how multiple, specialised latent representations can be
injected into the decoder's attention mechanism via low-rank operators. Our
empirical evaluation, carried out on natural language sentences and
mathematical expressions, reveals that the proposed end-to-end VAE architecture
can result in a better overall organisation of the latent space, alleviating
the information loss occurring in standard VAE setups, resulting in enhanced
performances on language modelling and downstream generation tasks.

通过多任务学习或双编码器架构将分布语义特征和句法结构的编码分离成异构的潜空间，整合基于图和序列模型，通过低秩算子将多个专门的潜在表示注入解码器的注意机制，结果显示所提出的端到端 VAE 架构可以得到更好的潜空间整理，减轻标准 VAE 架构中发生的信息丢失，从而改善语言建模和下游生成任务的性能。