Recently, both Contrastive Learning (CL) and Mask Image Modeling (MIM)
demonstrate that self-supervision is powerful to learn good representations.
However, naively combining them is far from success. In this paper, we start by
making the empirical observation that a naive joint optimization of CL and MIM
losses leads to conflicting gradient directions - more severe as the layers go
deeper. This motivates us to shift the paradigm from combining loss at the end,
to choosing the proper learning method per network layer. Inspired by
experimental observations, we find that MIM and CL are suitable to lower and
higher layers, respectively. We hence propose to combine them in a surprisingly
simple, "sequential cascade" fashion: early layers are first trained under one
MIM loss, on top of which latter layers continue to be trained under another CL
loss. The proposed Layer Grafted Pre-training learns good visual
representations that demonstrate superior label efficiency in downstream
applications, in particular yielding strong few-shot performance besides linear
evaluation. For instance, on ImageNet-1k, Layer Grafted Pre-training yields
65.5% Top-1 accuracy in terms of 1% few-shot learning with ViT-B/16, which
improves MIM and CL baselines by 14.4% and 2.1% with no bells and whistles. The
code is available at
this https URL

本文提出层级嫁接预训练 (Layer Grafted Pre-training) 方法，以笔者实验观察到的低层适合 MIM，高层适合 CL 为出发点，采用简单的 “顺序级联” 方式依次训练，旨在学习好的视觉表示。在 ImageNet-1k 数据集上，结合 ViT-B/16 能够取得 65.5% 的 1% few-shot 学习 Top-1 准确率，在无必要的改进的情况下提高了 MIM 和 CL 的基线水平 14.4% 和 2.1%。