Semantic sentence embeddings are usually supervisedly built minimizing
distances between pairs of embeddings of sentences labelled as semantically
similar by annotators. Since big labelled datasets are rare, in particular for
non-English languages, and expensive, recent studies focus on unsupervised
approaches that require not-paired input sentences. We instead propose a
language-independent approach to build large datasets of pairs of informal
texts weakly similar, without manual human effort, exploiting Twitter's
intrinsic powerful signals of relatedness: replies and quotes of tweets. We use
the collected pairs to train a Transformer model with triplet-like structures,
and we test the generated embeddings on Twitter NLP similarity tasks (PIT and
TURL) and STSb. We also introduce four new sentence ranking evaluation
benchmarks of informal texts, carefully extracted from the initial collections
of tweets, proving not only that our best model learns classical Semantic
Textual Similarity, but also excels on tasks where pairs of sentences are not
exact paraphrases. Ablation studies reveal how increasing the corpus size
influences positively the results, even at 2M samples, suggesting that bigger
collections of Tweets still do not contain redundant information about semantic
similarities.

本文提出了以 Twitter 数据集为基础的无监督 Transformer 模型，通过解析 Twitter 回复和引用语句来构建海量的非正式文本句子的成对数据集，从而训练生成的语义句子向量，不仅在经典语义文本相似度任务中表现出色，而且在非精确参数化相似句子任务上表现突出。

利用 Twitter 作为大语料库的来源，用于语义句子嵌入中的弱相似对

Exploiting Twitter as Source of Large Corpora of Weakly Similar Pairs  for Semantic Sentence Embeddings

Pre-trained language models have achieved huge success on a wide range of NLP
tasks. However, contextual representations from pre-trained models contain
entangled semantic and syntactic information, and therefore cannot be directly
used to derive useful semantic sentence embeddings for some tasks. Paraphrase
pairs offer an effective way of learning the distinction between semantics and
syntax, as they naturally share semantics and often vary in syntax. In this
work, we present ParaBART, a semantic sentence embedding model that learns to
disentangle semantics and syntax in sentence embeddings obtained by pre-trained
language models. ParaBART is trained to perform syntax-guided paraphrasing,
based on a source sentence that shares semantics with the target paraphrase,
and a parse tree that specifies the target syntax. In this way, ParaBART learns
disentangled semantic and syntactic representations from their respective
inputs with separate encoders. Experiments in English show that ParaBART
outperforms state-of-the-art sentence embedding models on unsupervised semantic
similarity tasks. Additionally, we show that our approach can effectively
remove syntactic information from semantic sentence embeddings, leading to
better robustness against syntactic variation on downstream semantic tasks.

ParaBART 是一个语义句子嵌入模型，使用基于源句子和目标释义之间语义相似性差异的语法指导解开基于预训练语言模型的嵌入中的语法和语义信息的工作。该方法在英语分析上表现出色，能有效的从语义句子嵌入中删除语法信息，提高对下游语义任务的鲁棒性。