In recent years, self-supervised learning (SSL) has emerged as a promising
approach for extracting valuable representations from unlabeled data. One
successful SSL method is contrastive learning, which aims to bring positive
examples closer while pushing negative examples apart. Many current contrastive
learning approaches utilize a parameterized projection head. Through a
combination of empirical analysis and theoretical investigation, we provide
insights into the internal mechanisms of the projection head and its
relationship with the phenomenon of dimensional collapse. Our findings
demonstrate that the projection head enhances the quality of representations by
performing contrastive loss in a projected subspace. Therefore, we propose an
assumption that only a subset of features is necessary when minimizing the
contrastive loss of a mini-batch of data. Theoretical analysis further suggests
that a sparse projection head can enhance generalization, leading us to
introduce SparseHead - a regularization term that effectively constrains the
sparsity of the projection head, and can be seamlessly integrated with any
self-supervised learning (SSL) approaches. Our experimental results validate
the effectiveness of SparseHead, demonstrating its ability to improve the
performance of existing contrastive methods.

自监督学习（SSL）是从无标签数据中提取有价值表示的一种有前途的方法，其中对比学习是一种成功的 SSL 方法，其目标是将正例拉近而将负例推开。通过实证分析和理论探究，我们揭示了投影头的内部机制及其与维度崩溃现象的关系，我们的研究结果表明，投影头通过在投影子空间中进行对比损失来提高表示的质量。因此，我们提出了一种假设，即在最小化一批数据的对比损失时，仅有部分特征是必要的。理论分析进一步表明，稀疏的投影头可以增强泛化能力，这导致我们引入了 SparseHead - 一种有效约束投影头稀疏性的正则化项，并可以无缝集成到任何自监督学习（SSL）方法中。我们的实验结果验证了 SparseHead 的有效性，证明了其改善现有对比方法性能的能力。