In this thesis, we develop methods to enhance the interpretability of recent representation learning techniques in natural language processing (NLP) while accounting for the unavailability of annotated data. We choose to leverage Variational Autoencoders (VAEs) due to their efficiency in relating observations to latent generative factors and their effectiveness in data-efficient learning and interpretable representation learning. As a first contribution, we identify and remove unnecessary components in the functioning scheme of semi-supervised VAEs making them faster, smaller and easier to design. Our second and main contribution is to use VAEs and Transformers to build two models with inductive bias to separate information in latent representations into understandable concepts without annotated data. The first model, Attention-Driven VAE (ADVAE), is able to separately represent and control information about syntactic roles in sentences. The second model, QKVAE, uses separate latent variables to form keys and values for its Transformer decoder and is able to separate syntactic and semantic information in its neural representations. In transfer experiments, QKVAE has competitive performance compared to supervised models and equivalent performance to a supervised model using 50K annotated samples. Additionally, QKVAE displays improved syntactic role disentanglement capabilities compared to ADVAE. Overall, we demonstrate that it is possible to enhance the interpretability of state-of-the-art deep learning architectures for language modeling with unannotated data in situations where text data is abundant but annotations are scarce.

提出使用变分自编码器和 Transformers 构建两种具有归纳偏置的模型，可将潜在表示中的信息分离成可理解的概念，其中 QKVAE 在转移实验中表现出竞争性能，并展示了明显的优化句法角色分离能力。

可解释的句子表示：变分自编码器和注意力机制