Contrastive learning has been gradually applied to learn high-quality
unsupervised sentence embedding. Among the previous un-supervised methods, the
latest state-of-the-art method, as far as we know, is unsupervised SimCSE
(unsup-SimCSE). Unsup-SimCSE uses the InfoNCE1loss function in the training
stage by pulling semantically similar sentences together and pushing apart
dis-similar ones.Theoretically, we expect to use larger batches in unsup-SimCSE
to get more adequate comparisons among samples and avoid overfitting. However,
increasing the batch size does not always lead to improvements, but instead
even lead to performance degradation when the batch size exceeds a threshold.
Through statistical observation, we find that this is probably due to the
introduction of low-confidence negative pairs after in-creasing the batch size.
To alleviate this problem, we introduce a simple smoothing strategy upon the
InfoNCE loss function, termedGaussian Smoothing InfoNCE
(GS-InfoNCE).Specifically, we add random Gaussian noise vectors as negative
samples, which act asa smoothing of the negative sample space.Though being
simple, the proposed smooth-ing strategy brings substantial improvements to
unsup-SimCSE. We evaluate GS-InfoNCEon the standard semantic text similarity
(STS)task. GS-InfoNCE outperforms the state-of-the-art unsup-SimCSE by an
average Spear-man correlation of 1.38%, 0.72%, 1.17% and0.28% on the base of
BERT-base, BERT-large,RoBERTa-base and RoBERTa-large, respectively.

该论文提出了一种简单有效的平滑策略 (Gaussian Smoothing InfoNCE)，以解决在使用较大 batch size 时信息噪声过多导致性能下降的问题，在标准语义文本相似度任务 (STS) 上表现超过了目前最先进的无监督学习方法 (unsup-SimCSE)