In this thesis, we develop methods to enhance the interpretability of recent
representation learning techniques in natural language processing (NLP) while
accounting for the unavailability of annotated data. We choose to leverage
Variational Autoencoders (VAEs) due to their efficiency in relating
observations to latent generative factors and their effectiveness in
data-efficient learning and interpretable representation learning. As a first
contribution, we identify and remove unnecessary components in the functioning
scheme of semi-supervised VAEs making them faster, smaller and easier to
design. Our second and main contribution is to use VAEs and Transformers to
build two models with inductive bias to separate information in latent
representations into understandable concepts without annotated data. The first
model, Attention-Driven VAE (ADVAE), is able to separately represent and
control information about syntactic roles in sentences. The second model,
QKVAE, uses separate latent variables to form keys and values for its
Transformer decoder and is able to separate syntactic and semantic information
in its neural representations. In transfer experiments, QKVAE has competitive
performance compared to supervised models and equivalent performance to a
supervised model using 50K annotated samples. Additionally, QKVAE displays
improved syntactic role disentanglement capabilities compared to ADVAE.
Overall, we demonstrate that it is possible to enhance the interpretability of
state-of-the-art deep learning architectures for language modeling with
unannotated data in situations where text data is abundant but annotations are
scarce.

提出使用变分自编码器和 Transformers 构建两种具有归纳偏置的模型，可将潜在表示中的信息分离成可理解的概念，其中 QKVAE 在转移实验中表现出竞争性能，并展示了明显的优化句法角色分离能力。