Self-supervised representation learning maps high-dimensional data into a
meaningful embedding space, where samples of similar semantic contents are
close to each other. Most of the recent representation learning methods
maximize cosine similarity or minimize the distance between the embedding
features of different views from the same sample usually on the $l2$ normalized
unit hypersphere. To prevent the trivial solutions that all samples have the
same embedding feature, various techniques have been developed, such as
contrastive learning, stop gradient, variance and covariance regularization,
etc. In this study, we propose MUlti-Segmental Informational Coding (MUSIC) for
self-supervised representation learning. MUSIC divides the embedding feature
into multiple segments that discriminatively partition samples into different
semantic clusters and different segments focus on different partition
principles. Information theory measurements are directly used to optimize MUSIC
and theoretically guarantee trivial solutions are avoided. MUSIC does not
depend on commonly used techniques, such as memory bank or large batches,
asymmetry networks, gradient stopping, momentum weight updating, etc, making
the training framework flexible. Our experiments demonstrate that MUSIC
achieves better results than most related Barlow Twins and VICReg methods on
ImageNet classification with linear probing, and requires neither deep
projectors nor large feature dimensions. Code will be made available.

本文提出了 MUlti-Segmental Informational Coding（MUSIC）方法，使用信息理论测量直接优化划分样本到不同语义聚类的多个划分部分，以避免最小解，证明了 MUSIC 比最相关的 Barlow Twins 和 VICReg 方法在 ImageNet 分类方面线性探测时具有更好的结果。