Recent self-supervised methods for image representation learning are based on maximizing the agreement between embedding vectors from different views of the same image. A trivial solution is obtained when the encoder outputs constant vectors. This collapse problem is often avoided through implicit biases in the learning architecture, that often lack a clear justification or interpretation. In this paper, we introduce VICReg (Variance-Invariance-Covariance Regularization), a method that explicitly avoids the collapse problem with a simple regularization term on the variance of the embeddings along each dimension individually. VICReg combines the variance term with a decorrelation mechanism based on redundancy reduction and covariance regularization, and achieves results on par with the state of the art on several downstream tasks. In addition, we show that incorporating our new variance term into other methods helps stabilize the training and leads to performance improvements.

本文介绍了一种简单、易于解释的方法 VICReg，结合了方差约束、冗余降维和协方差正则化等策略，可以解决自我监督学习中模型输出恒定向量的问题，并能在图像表征学习等多个下游任务中达到与现有方法相当的性能水平。

VICReg: 自监督学习的方差不变协方差正则化