Supervised contrastive loss (SCL) is a competitive and often superior
alternative to the cross-entropy (CE) loss for classification. In this paper we
ask: what differences in the learning process occur when the two different loss
functions are being optimized? To answer this question, our main finding is
that the geometry of embeddings learned by SCL forms an orthogonal frame (OF)
regardless of the number of training examples per class. This is in contrast to
the CE loss, for which previous work has shown that it learns embeddings
geometries that are highly dependent on the class sizes. We arrive at our
finding theoretically, by proving that the global minimizers of an
unconstrained features model with SCL loss and entry-wise non-negativity
constraints form an OF. We then validate the model's prediction by conducting
experiments with standard deep-learning models on benchmark vision datasets.
Finally, our analysis and experiments reveal that the batching scheme chosen
during SCL training plays a critical role in determining the quality of
convergence to the OF geometry. This finding motivates a simple algorithm
wherein the addition of a few binding examples in each batch significantly
speeds up the occurrence of the OF geometry.

本文通过实验证明：监督对比损失学习到的嵌入空间的几何结构形成正交框架，而交叉熵损失学习到的嵌入空间的几何结构高度依赖于类大小；同时，通过选取不同的数据分组算法，加快正交框架的出现，为监督学习提供一种新的训练策略