The mechanisms behind the success of multi-view self-supervised learning
(MVSSL) are not yet fully understood. Contrastive MVSSL methods have been
studied through the lens of InfoNCE, a lower bound of the Mutual Information
(MI). However, the relation between other MVSSL methods and MI remains unclear.
We consider a different lower bound on the MI consisting of an entropy and a
reconstruction term (ER), and analyze the main MVSSL families through its lens.
Through this ER bound, we show that clustering-based methods such as
DeepCluster and SwAV maximize the MI. We also re-interpret the mechanisms of
distillation-based approaches such as BYOL and DINO, showing that they
explicitly maximize the reconstruction term and implicitly encourage a stable
entropy, and we confirm this empirically. We show that replacing the objectives
of common MVSSL methods with this ER bound achieves competitive performance,
while making them stable when training with smaller batch sizes or smaller
exponential moving average (EMA) coefficients.
Github repo: this https URL

多视角自监督学习的成功机制尚未完全了解，本文通过熵和重构项 (ER) 的下界进行分析，发现基于聚类的方法最大化了互信息 (MI)，而基于蒸馏的方法则显式地最大化了重构项并隐式地鼓励稳定熵，通过用 ER 下界替换常见 MVSSL 方法的目标，实现了竞争性的性能，并在小批量大小或小指数移动平均 (EMA) 系数下保持稳定。