Pre-trained language models have achieved huge success on a wide range of NLP
tasks. However, contextual representations from pre-trained models contain
entangled semantic and syntactic information, and therefore cannot be directly
used to derive useful semantic sentence embeddings for some tasks. Paraphrase
pairs offer an effective way of learning the distinction between semantics and
syntax, as they naturally share semantics and often vary in syntax. In this
work, we present ParaBART, a semantic sentence embedding model that learns to
disentangle semantics and syntax in sentence embeddings obtained by pre-trained
language models. ParaBART is trained to perform syntax-guided paraphrasing,
based on a source sentence that shares semantics with the target paraphrase,
and a parse tree that specifies the target syntax. In this way, ParaBART learns
disentangled semantic and syntactic representations from their respective
inputs with separate encoders. Experiments in English show that ParaBART
outperforms state-of-the-art sentence embedding models on unsupervised semantic
similarity tasks. Additionally, we show that our approach can effectively
remove syntactic information from semantic sentence embeddings, leading to
better robustness against syntactic variation on downstream semantic tasks.

ParaBART 是一个语义句子嵌入模型，使用基于源句子和目标释义之间语义相似性差异的语法指导解开基于预训练语言模型的嵌入中的语法和语义信息的工作。该方法在英语分析上表现出色，能有效的从语义句子嵌入中删除语法信息，提高对下游语义任务的鲁棒性。

基于预训练语言模型的句子嵌入中语义与句法的解耦

Disentangling Semantics and Syntax in Sentence Embeddings with  Pre-trained Language Models

Semantic sentence embedding models encode natural language sentences into
vectors, such that closeness in embedding space indicates closeness in the
semantics between the sentences. Bilingual data offers a useful signal for
learning such embeddings: properties shared by both sentences in a translation
pair are likely semantic, while divergent properties are likely stylistic or
language-specific. We propose a deep latent variable model that attempts to
perform source separation on parallel sentences, isolating what they have in
common in a latent semantic vector, and explaining what is left over with
language-specific latent vectors. Our proposed approach differs from past work
on semantic sentence encoding in two ways. First, by using a variational
probabilistic framework, we introduce priors that encourage source separation,
and can use our model's posterior to predict sentence embeddings for
monolingual data at test time. Second, we use high-capacity transformers as
both data generating distributions and inference networks -- contrasting with
most past work on sentence embeddings. In experiments, our approach
substantially outperforms the state-of-the-art on a standard suite of
unsupervised semantic similarity evaluations. Further, we demonstrate that our
approach yields the largest gains on more difficult subsets of these
evaluations where simple word overlap is not a good indicator of similarity.

本文提出了一种基于变分概率框架的深度潜变量模型，使用双语数据，利用拥有共性的语义信息与特性进行训练，使模型可以将并行语句进行源分离，得到隐含的语义向量，进而对单语数据进行预测。实验证明，该方法在无监督语义相似性评估方面的表现明显优于现有方法，并且还能在不适合使用简单词语重叠作为相似性指标的更难评估子集中发挥最大的作用。