Previous unsupervised sentence embedding studies have focused on data
augmentation methods such as dropout masking and rule-based sentence
transformation methods. However, these approaches have a limitation of
controlling the fine-grained semantics of augmented views of a sentence. This
results in inadequate supervision signals for capturing a semantic similarity
of similar sentences. In this work, we found that using neighbor sentences
enables capturing a more accurate semantic similarity between similar
sentences. Based on this finding, we propose RankEncoder, which uses relations
between an input sentence and sentences in a corpus for training unsupervised
sentence encoders. We evaluate RankEncoder from three perspectives: 1) the
semantic textual similarity performance, 2) the efficacy on similar sentence
pairs, and 3) the universality of RankEncoder. Experimental results show that
RankEncoder achieves 80.07% Spearman's correlation, a 1.1% absolute improvement
compared to the previous state-of-the-art performance. The improvement is even
more significant, a 1.73% improvement, on similar sentence pairs. Also, we
demonstrate that RankEncoder is universally applicable to existing unsupervised
sentence encoders.

本文提出了一种基于相邻句子的无监督句子编码器 RankEncoder，利用相邻句子约束进行训练，在语义文本相似度性能、相似句子对的有效性和普适性等方面都较好地表现，达到了 80.07% 的斯皮尔曼等级相关性，相较于最先进技术性能有 1.1% 的绝对提升，并且在相似句子对方面提升更为显着，为 1.73%，同时证明了 RankEncoder 可以普遍适用于现有的无监督句子编码器。

增强排名的无监督句子表示学习

Ranking-Enhanced Unsupervised Sentence Representation Learning

Unsupervised sentence embedding aims to obtain the most appropriate embedding
for a sentence to reflect its semantic. Contrastive learning has been
attracting developing attention. For a sentence, current models utilize diverse
data augmentation methods to generate positive samples, while consider other
independent sentences as negative samples. Then they adopt InfoNCE loss to pull
the embeddings of positive pairs gathered, and push those of negative pairs
scattered. Although these models have made great progress on sentence
embedding, we argue that they may suffer from feature suppression. The models
fail to distinguish and decouple textual similarity and semantic similarity.
And they may overestimate the semantic similarity of any pairs with similar
textual regardless of the actual semantic difference between them. This is
because positive pairs in unsupervised contrastive learning come with similar
and even the same textual through data augmentation. To alleviate feature
suppression, we propose contrastive learning for unsupervised sentence
embedding with soft negative samples (SNCSE). Soft negative samples share
highly similar textual but have surely and apparently different semantic with
the original samples. Specifically, we take the negation of original sentences
as soft negative samples, and propose Bidirectional Margin Loss (BML) to
introduce them into traditional contrastive learning framework, which merely
involves positive and negative samples. Our experimental results show that
SNCSE can obtain state-of-the-art performance on semantic textual similarity
(STS) task with average Spearman's correlation coefficient of 78.97% on
BERTbase and 79.23% on RoBERTabase. Besides, we adopt rank-based error analysis
method to detect the weakness of SNCSE for future study.

本文通过引入软负样本和双向边缘损失来改善当前无监督对比学习中的特征抑制问题，在语义文本相似度任务上取得了最优结果。

SNCSE: 无监督句子嵌入的对比学习与软负样本

SNCSE: Contrastive Learning for Unsupervised Sentence Embedding with Soft Negative Samples

Contrastive learning has been gradually applied to learn high-quality
unsupervised sentence embedding. Among the previous un-supervised methods, the
latest state-of-the-art method, as far as we know, is unsupervised SimCSE
(unsup-SimCSE). Unsup-SimCSE uses the InfoNCE1loss function in the training
stage by pulling semantically similar sentences together and pushing apart
dis-similar ones.Theoretically, we expect to use larger batches in unsup-SimCSE
to get more adequate comparisons among samples and avoid overfitting. However,
increasing the batch size does not always lead to improvements, but instead
even lead to performance degradation when the batch size exceeds a threshold.
Through statistical observation, we find that this is probably due to the
introduction of low-confidence negative pairs after in-creasing the batch size.
To alleviate this problem, we introduce a simple smoothing strategy upon the
InfoNCE loss function, termedGaussian Smoothing InfoNCE
(GS-InfoNCE).Specifically, we add random Gaussian noise vectors as negative
samples, which act asa smoothing of the negative sample space.Though being
simple, the proposed smooth-ing strategy brings substantial improvements to
unsup-SimCSE. We evaluate GS-InfoNCEon the standard semantic text similarity
(STS)task. GS-InfoNCE outperforms the state-of-the-art unsup-SimCSE by an
average Spear-man correlation of 1.38%, 0.72%, 1.17% and0.28% on the base of
BERT-base, BERT-large,RoBERTa-base and RoBERTa-large, respectively.

该论文提出了一种简单有效的平滑策略 (Gaussian Smoothing InfoNCE)，以解决在使用较大 batch size 时信息噪声过多导致性能下降的问题，在标准语义文本相似度任务 (STS) 上表现超过了目前最先进的无监督学习方法 (unsup-SimCSE)