Bootstrap Your Own Latent (BYOL) is a self-supervised learning approach for
image representation. From an augmented view of an image, BYOL trains an online
network to predict a target network representation of a different augmented
view of the same image. Unlike contrastive methods, BYOL does not explicitly
use a repulsion term built from negative pairs in its training objective. Yet,
it avoids collapse to a trivial, constant representation. Thus, it has recently
been hypothesized that batch normalization (BN) is critical to prevent collapse
in BYOL. Indeed, BN flows gradients across batch elements, and could leak
information about negative views in the batch, which could act as an implicit
negative (contrastive) term. However, we experimentally show that replacing BN
with a batch-independent normalization scheme (namely, a combination of group
normalization and weight standardization) achieves performance comparable to
vanilla BYOL ($73.9\%$ vs. $74.3\%$ top-1 accuracy under the linear evaluation
protocol on ImageNet with ResNet-$50$). Our finding disproves the hypothesis
that the use of batch statistics is a crucial ingredient for BYOL to learn
useful representations.

本研究提出 Bootstrap Your Own Latent (BYOL) 作为一种自监督学习方法，用于图像表示学习，并探讨与批归一化 (batch normalization) 的关系。实验结果表明，使用批独立归一化可以取得与传统 BYOL 相当的性能，未必需要批归一化。