We show that the gradient of the cosine similarity between two points goes to zero in two under-explored settings: (1) if a point has large magnitude or (2) if the points are on opposite ends of the latent space. Counterintuitively, we prove that optimizing the cosine similarity between points forces them to grow in magnitude. Thus, (1) is unavoidable in practice. We then observe that these derivations are extremely general -- they hold across deep learning architectures and for many of the standard self-supervised learning (SSL) loss functions. This leads us to propose cut-initialization: a simple change to network initialization that helps all studied SSL methods converge faster.

我们展示了在两种未被充分探索的情况下，两点之间的余弦相似度的梯度趋近于零：（1）如果一个点的幅度很大，或者（2）如果这两个点位于潜在空间的两端。令人意想不到的是，我们证明，优化点之间的余弦相似度会导致它们的幅度增加。因此，（1）在实践中是不可避免的。然后，我们观察到这些推导非常通用，适用于深度学习架构和许多标准的自监督学习（SSL）损失函数。这引导我们提出切割初始化（cut-initialization）：一种对网络初始化进行简单更改的方法，有助于所有研究的自监督学习方法更快地收敛。

余弦相似度损失的隐蔽陷阱