This study focuses on addressing the instability issues prevalent in
contrastive learning, specifically examining the InfoNCE loss function and its
derivatives. We reveal a critical observation that these loss functions exhibit
a restrictive behavior, leading to a convergence phenomenon where embeddings
tend to merge into a singular point. This "over-fusion" effect detrimentally
affects classification accuracy in subsequent supervised-learning tasks.
Through theoretical analysis, we demonstrate that embeddings, when equalized or
confined to a rank-1 linear subspace, represent a local minimum for InfoNCE. In
response to this challenge, our research introduces an innovative strategy that
leverages the same or fewer labeled data than typically used in the fine-tuning
phase. The loss we proposed, Orthonormal Anchor Regression Loss, is designed to
disentangle embedding clusters, significantly enhancing the distinctiveness of
each embedding while simultaneously ensuring their aggregation into dense,
well-defined clusters. Our method demonstrates remarkable improvements with
just a fraction of the conventional label requirements, as evidenced by our
results on CIFAR10 and CIFAR100 datasets.

本研究旨在解决对比学习中普遍存在的不稳定问题，具体研究 InfoNCE 损失函数及其导数。我们揭示了这些损失函数表现出的一种限制行为，导致嵌入倾向于融合成一个奇异点的收敛现象，从而对后续的有监督学习任务中的分类准确性产生不利影响。通过理论分析，我们证明当嵌入被等化或限制在一个一维线性子空间时，它们代表了 InfoNCE 的一个局部最小值。我们的研究提出了一种创新策略，利用与精调阶段中通常使用的标记数据相同或更少的数据。我们提出的损失函数，正交锚回归损失，旨在解开嵌入聚类，显著增强每个嵌入的独特性，同时确保它们聚合到密集的、明确定义的聚类中。我们的方法在 CIFAR10 和 CIFAR100 数据集上通过仅使用传统标记要求的一小部分数据取得了显著改善，这一点可以从实验结果中得到证明。