Pre-trained language models have achieved huge success on a wide range of NLP
tasks. However, contextual representations from pre-trained models contain
entangled semantic and syntactic information, and therefore cannot be directly
used to derive useful semantic sentence embeddings for some tasks. Paraphrase
pairs offer an effective way of learning the distinction between semantics and
syntax, as they naturally share semantics and often vary in syntax. In this
work, we present ParaBART, a semantic sentence embedding model that learns to
disentangle semantics and syntax in sentence embeddings obtained by pre-trained
language models. ParaBART is trained to perform syntax-guided paraphrasing,
based on a source sentence that shares semantics with the target paraphrase,
and a parse tree that specifies the target syntax. In this way, ParaBART learns
disentangled semantic and syntactic representations from their respective
inputs with separate encoders. Experiments in English show that ParaBART
outperforms state-of-the-art sentence embedding models on unsupervised semantic
similarity tasks. Additionally, we show that our approach can effectively
remove syntactic information from semantic sentence embeddings, leading to
better robustness against syntactic variation on downstream semantic tasks.

ParaBART 是一个语义句子嵌入模型，使用基于源句子和目标释义之间语义相似性差异的语法指导解开基于预训练语言模型的嵌入中的语法和语义信息的工作。该方法在英语分析上表现出色，能有效的从语义句子嵌入中删除语法信息，提高对下游语义任务的鲁棒性。