This paper aims to improve contrastive learning for sentence embeddings from
two perspectives: handling dropout noise and addressing feature corruption.
Specifically, for the first perspective, we identify that the dropout noise
from negative pairs affects the model's performance. Therefore, we propose a
simple yet effective method to deal with such type of noise. Secondly, we
pinpoint the rank bottleneck of current solutions to feature corruption and
propose a dimension-wise contrastive learning objective to address this issue.
Both proposed methods are generic and can be applied to any contrastive
learning based models for sentence embeddings. Experimental results on standard
benchmarks demonstrate that combining both proposed methods leads to a gain of
1.8 points compared to the strong baseline SimCSE configured with BERT base.
Furthermore, applying the proposed method to DiffCSE, another strong
contrastive learning based baseline, results in a gain of 1.4 points.

本文从处理 dropout 噪声和解决特征损坏两个角度，提出了两种改进对比学习的方法，均能提高句子嵌入的性能，实验证明，将两种方法结合使用，相较于基于 BERT 基础的强基线模型 SimCSE，性能提升了 1.8 个点；同时，这些方法还能适用于 DiffCSE 等其他有着强鲁棒性的模型。