Contrastive-learning-based methods have dominated sentence representation
learning. These methods regularize the representation space by pulling similar
sentence representations closer and pushing away the dissimilar ones and have
been proven effective in various NLP tasks, e.g., semantic textual similarity
(STS) tasks. However, it is challenging for these methods to learn fine-grained
semantics as they only learn from the inter-sentence perspective, i.e., their
supervision signal comes from the relationship between data samples. In this
work, we propose a novel denoising objective that inherits from another
perspective, i.e., the intra-sentence perspective. By introducing both discrete
and continuous noise, we generate noisy sentences and then train our model to
restore them to their original form. Our empirical evaluations demonstrate that
this approach delivers competitive results on both semantic textual similarity
(STS) and a wide range of transfer tasks, standing up well in comparison to
contrastive-learning-based methods. Notably, the proposed intra-sentence
denoising objective complements existing inter-sentence contrastive
methodologies and can be integrated with them to further enhance performance.
Our code is available at this https URL

通过引入离散和连续噪声，我们提出了一种新的去噪目标，通过训练模型将噪声句子恢复为其原始形式，从而在语义文本相似性和广泛的迁移任务中实现了有竞争力的结果。