Pre-trained language models have proven their unique powers in capturing
implicit language features. However, most pre-training approaches focus on the
word-level training objective, while sentence-level objectives are rarely
studied. In this paper, we propose Contrastive LEArning for sentence
Representation (CLEAR), which employs multiple sentence-level augmentation
strategies in order to learn a noise-invariant sentence representation. These
augmentations include word and span deletion, reordering, and substitution.
Furthermore, we investigate the key reasons that make contrastive learning
effective through numerous experiments. We observe that different sentence
augmentations during pre-training lead to different performance improvements on
various downstream tasks. Our approach is shown to outperform multiple existing
methods on both SentEval and GLUE benchmarks.

本研究提出了 CLEAR 方法，利用多种句子级别数据增强策略来学习一种噪声不变的句子表示。通过多个实验发现数据增强的不同方法会在下游任务中带来不同的性能提升，并且本方法在 SentEval 和 GLUE 基准测试中的表现优于多种现有方法。