Self-supervised learning allows AI systems to learn effective representations
from large amounts of data using tasks that do not require costly labeling.
Mode collapse, i.e., the model producing identical representations for all
inputs, is a central problem to many self-supervised learning approaches,
making self-supervised tasks, such as matching distorted variants of the
inputs, ineffective. In this article, we argue that a straightforward
application of information maximization among alternative latent
representations of the same input naturally solves the collapse problem and
achieves competitive empirical results. We propose a self-supervised learning
method, CorInfoMax, that uses a second-order statistics-based mutual
information measure that reflects the level of correlation among its arguments.
Maximizing this correlative information measure between alternative
representations of the same input serves two purposes: (1) it avoids the
collapse problem by generating feature vectors with non-degenerate covariances;
(2) it establishes relevance among alternative representations by increasing
the linear dependence among them. An approximation of the proposed information
maximization objective simplifies to a Euclidean distance-based objective
function regularized by the log-determinant of the feature covariance matrix.
The regularization term acts as a natural barrier against feature space
degeneracy. Consequently, beyond avoiding complete output collapse to a single
point, the proposed approach also prevents dimensional collapse by encouraging
the spread of information across the whole feature space. Numerical experiments
demonstrate that CorInfoMax achieves better or competitive performance results
relative to the state-of-the-art SSL approaches.

本文介绍一种基于互信息的 self-supervised 方法 CorInfoMax，该方法通过最大化高斯混合分布下的互信息解决了 mode collapse 和 dimensional collapse 问题。