While contrastive approaches of self-supervised learning (SSL) learn
representations by minimizing the distance between two augmented views of the
same data point (positive pairs) and maximizing views from different data
points (negative pairs), recent \emph{non-contrastive} SSL (e.g., BYOL and
SimSiam) show remarkable performance {\it without} negative pairs, with an
extra learnable predictor and a stop-gradient operation. A fundamental question
arises: why do these methods not collapse into trivial representations? We
answer this question via a simple theoretical study and propose a novel
approach, DirectPred, that \emph{directly} sets the linear predictor based on
the statistics of its inputs, without gradient training. On ImageNet, it
performs comparably with more complex two-layer non-linear predictors that
employ BatchNorm and outperforms a linear predictor by $2.5\%$ in 300-epoch
training (and $5\%$ in 60-epoch). DirectPred is motivated by our theoretical
study of the nonlinear learning dynamics of non-contrastive SSL in simple
linear networks. Our study yields conceptual insights into how non-contrastive
SSL methods learn, how they avoid representational collapse, and how multiple
factors, like predictor networks, stop-gradients, exponential moving averages,
and weight decay all come into play. Our simple theory recapitulates the
results of real-world ablation studies in both STL-10 and ImageNet. Code is
released this https URL

在自监督学习中，我们提出一种名为 DirectPred 的新方法，该方法直接基于输入的统计信息而非梯度训练来设置线性预测器。在 ImageNet 上，该方法的表现与采用 BatchNorm 的更复杂的两层非线性预测器相当，且在 300 个 epoch 的训练中优于线性预测器 2.5％（在 60 个 epoch 中达到 5％）。同时，通过研究非对比性 SSL 在简单线性网络中的非线性学习动态，我们为这些方法的学习和避免表征崩溃提供了理论洞察力。