The information noise-contrastive estimation (InfoNCE) loss function provides
the basis of many self-supervised deep learning methods due to its strong
empirical results and theoretic motivation. Previous work suggests a supervised
contrastive (SupCon) loss to extend InfoNCE to learn from available class
labels. This SupCon loss has been widely-used due to reports of good empirical
performance. However, in this work we suggest that the specific SupCon loss
formulated by prior work has questionable theoretic justification, because it
can encourage images from the same class to repel one another in the learned
embedding space. This problematic behavior gets worse as the number of inputs
sharing one class label increases. We propose the Supervised InfoNCE REvisited
(SINCERE) loss as a remedy. SINCERE is a theoretically justified solution for a
supervised extension of InfoNCE that never causes images from the same class to
repel one another. We further show that minimizing our new loss is equivalent
to maximizing a bound on the KL divergence between class conditional embedding
distributions. We compare SINCERE and SupCon losses in terms of learning
trajectories during pretraining and in ultimate linear classifier performance
after finetuning. Our proposed SINCERE loss better separates embeddings from
different classes during pretraining while delivering competitive accuracy.

通过改进 InfoNCE 的 Supervised InfoNCE REvisited（SINCERE）损失函数，我们提出了一种理论上合理的自监督深度学习方法，可以有效防止同类图像在嵌入空间中互相排斥，从而提高预训练过程中的类别嵌入分离性和最终线性分类器的性能。