Unsupervised sentence representation learning aims to transform input
sentences into fixed-length vectors enriched with intricate semantic
information while obviating the reliance on labeled data. Recent progress
within this field, propelled by contrastive learning and prompt engineering,
has significantly bridged the gap between unsupervised and supervised
strategies. Nonetheless, the potential utilization of Chain-of-Thought, remains
largely untapped within this trajectory. To unlock latent capabilities within
pre-trained models, such as BERT, we propose a two-stage approach for sentence
representation: comprehension and summarization. Subsequently, the output of
the latter phase is harnessed as the vectorized representation of the input
sentence. For further performance enhancement, we meticulously refine both the
contrastive learning loss function and the template denoising technique for
prompt engineering. Rigorous experimentation substantiates our method,
CoT-BERT, transcending a suite of robust baselines without necessitating other
text representation models or external databases.

利用无监督的句子表示学习方法，通过对输入句子进行转换，得到富含复杂语义信息的定长向量表示，消除了对标注数据的依赖。本文提出了一种两阶段的方法，通过理解和总结对预训练模型（如 BERT）的潜在能力进行开发，并通过精细调整对比学习损失函数和模板去噪技术来进一步提高性能。实验证明，所提出的 CoT-BERT 方法在无需其他文本表示模型或外部数据库的情况下超越了一系列强大的基准模型。

CoT-BERT: 通过思维链增强无监督句子表示

CoT-BERT: Enhancing Unsupervised Sentence Representation through  Chain-of-Thought

Pre-trained Language Model (PLM) is nowadays the mainstay of Unsupervised
Sentence Representation Learning (USRL). However, PLMs are sensitive to the
frequency information of words from their pre-training corpora, resulting in
anisotropic embedding space, where the embeddings of high-frequency words are
clustered but those of low-frequency words disperse sparsely. This anisotropic
phenomenon results in two problems of similarity bias and information bias,
lowering the quality of sentence embeddings. To solve the problems, we
fine-tune PLMs by leveraging the frequency information of words and propose a
novel USRL framework, namely Sentence Representation Learning with
Frequency-induced Adversarial tuning and Incomplete sentence filtering
(SLT-FAI). We calculate the word frequencies over the pre-training corpora of
PLMs and assign words thresholding frequency labels. With them, (1) we
incorporate a similarity discriminator used to distinguish the embeddings of
high-frequency and low-frequency words, and adversarially tune the PLM with it,
enabling to achieve uniformly frequency-invariant embedding space; and (2) we
propose a novel incomplete sentence detection task, where we incorporate an
information discriminator to distinguish the embeddings of original sentences
and incomplete sentences by randomly masking several low-frequency words,
enabling to emphasize the more informative low-frequency words. Our SLT-FAI is
a flexible and plug-and-play framework, and it can be integrated with existing
USRL techniques. We evaluate SLT-FAI with various backbones on benchmark
datasets. Empirical results indicate that SLT-FAI can be superior to the
existing USRL baselines. Our code is released in
https://github.com/wangbing1416/SLT-FAI.

提出了一种新的无监督句子表示学习（USRL）框架，称为带有频率诱导的对抗微调和不完整句子过滤器的句子表示学习（SLT-FAI），通过利用单词频率信息来解决预先训练语言模型（PLM）中单词频率信息敏感性带来的相似偏差和信息偏差问题，此框架灵活易用并能优于现有的 USRL 基线。