We present Transformation Invariance and Covariance Contrast (TiCo) for
self-supervised visual representation learning. Similar to other recent
self-supervised learning methods, our method is based on maximizing the
agreement among embeddings of different distorted versions of the same image,
which pushes the encoder to produce transformation invariant representations.
To avoid the trivial solution where the encoder generates constant vectors, we
regularize the covariance matrix of the embeddings from different images by
penalizing low rank solutions. By jointly minimizing the transformation
invariance loss and covariance contrast loss, we get an encoder that is able to
produce useful representations for downstream tasks. We analyze our method and
show that it can be viewed as a variant of MoCo with an implicit memory bank of
unlimited size at no extra memory cost. This makes our method perform better
than alternative methods when using small batch sizes. TiCo can also be seen as
a modification of Barlow Twins. By connecting the contrastive and
redundancy-reduction methods together, TiCo gives us new insights into how
joint embedding methods work.

TiCo 是一种基于自监督学习的视觉表示学习方法，通过最大化同一图像不同畸变版本的嵌入之间的一致性来使编码器产生具有变换不变性的表示，并通过正则化来避免平凡解。通过共同最小化转换不变性损失和协方差对比损失，获得能够产生有用表示用于下游任务的编码器，这种方法提供了一种视觉嵌入方法的新思路。