Several prior studies have suggested that word frequency biases can cause the
Bert model to learn indistinguishable sentence embeddings. Contrastive learning
schemes such as SimCSE and ConSERT have already been adopted successfully in
unsupervised sentence embedding to improve the quality of embeddings by
reducing this bias. However, these methods still introduce new biases such as
sentence length bias and false negative sample bias, that hinders model's
ability to learn more fine-grained semantics. In this paper, we reexamine the
challenges of contrastive sentence embedding learning from a debiasing
perspective and argue that effectively eliminating the influence of various
biases is crucial for learning high-quality sentence embeddings. We think all
those biases are introduced by simple rules for constructing training data in
contrastive learning and the key for contrastive learning sentence embedding is
to mimic the distribution of training data in supervised machine learning in
unsupervised way. We propose a novel contrastive framework for sentence
embedding, termed DebCSE, which can eliminate the impact of these biases by an
inverse propensity weighted sampling method to select high-quality positive and
negative pairs according to both the surface and semantic similarity between
sentences. Extensive experiments on semantic textual similarity (STS)
benchmarks reveal that DebCSE significantly outperforms the latest
state-of-the-art models with an average Spearman's correlation coefficient of
80.33% on BERTbase.

本文研究了对比学习中存在的偏见问题，提出了一种名为 DebCSE 的新型对比框架，通过逆倾向加权抽样方法选择高质量的正负样本对，以消除各种偏见的影响，从而提高句子嵌入的质量。在语义文本相似性基准测试中，DebCSE 的平均 Spearman 相关系数为 80.33%，显著优于最新的最先进模型。