Contrastive learning-based methods, such as unsup-SimCSE, have achieved
state-of-the-art (SOTA) performances in learning unsupervised sentence
embeddings. However, in previous studies, each embedding used for contrastive
learning only derived from one sentence instance, and we call these embeddings
instance-level embeddings. In other words, each embedding is regarded as a
unique class of its own, whichmay hurt the generalization performance. In this
study, we propose IS-CSE (instance smoothing contrastive sentence embedding) to
smooth the boundaries of embeddings in the feature space. Specifically, we
retrieve embeddings from a dynamic memory buffer according to the semantic
similarity to get a positive embedding group. Then embeddings in the group are
aggregated by a self-attention operation to produce a smoothed instance
embedding for further analysis. We evaluate our method on standard semantic
text similarity (STS) tasks and achieve an average of 78.30%, 79.47%, 77.73%,
and 79.42% Spearman's correlation on the base of BERT-base, BERT-large,
RoBERTa-base, and RoBERTa-large respectively, a 2.05%, 1.06%, 1.16% and 0.52%
improvement compared to unsup-SimCSE.

本文提出了 IS-CSE 方法用于平滑特征空间中的嵌入，实现句子嵌入的无监督学习，进而在标准的语义文本相似度（STS）任务中取得更好的结果。