The latest advancements in unsupervised learning of sentence embeddings
predominantly involve employing contrastive learning-based (CL-based)
fine-tuning over pre-trained language models. In this study, we analyze the
latest sentence embedding methods by adopting representation rank as the
primary tool of analysis. We first define Phase 1 and Phase 2 of fine-tuning
based on when representation rank peaks. Utilizing these phases, we conduct a
thorough analysis and obtain essential findings across key aspects, including
alignment and uniformity, linguistic abilities, and correlation between
performance and rank. For instance, we find that the dynamics of the key
aspects can undergo significant changes as fine-tuning transitions from Phase 1
to Phase 2. Based on these findings, we experiment with a rank reduction (RR)
strategy that facilitates rapid and stable fine-tuning of the latest CL-based
methods. Through empirical investigations, we showcase the efficacy of RR in
enhancing the performance and stability of five state-of-the-art sentence
embedding methods.

本研究通过使用对比学习的 fine-tuning 方法并以表示排名作为主要分析工具，分析了最新的句子嵌入方法。通过定义 fine-tuning 的两个阶段，并利用这些阶段进行全面分析，我们得出了关于对齐和一致性、语言能力以及性能和排名之间相关性的重要发现。基于这些发现，我们提出了一种排名降低（RR）策略，以促进最新的对比学习方法的快速和稳定的 fine-tuning，并通过实证研究展示了 RR 在提高五种最先进句子嵌入方法的性能和稳定性方面的有效性。