Masked image modeling (MIM) learns representations with remarkably good
fine-tuning performances, overshadowing previous prevalent pre-training
approaches such as image classification, instance contrastive learning, and
image-text alignment. In this paper, we show that the inferior fine-tuning
performance of these pre-training approaches can be significantly improved by a
simple post-processing in the form of feature distillation (FD). The feature
distillation converts the old representations to new representations that have
a few desirable properties just like those representations produced by MIM.
These properties, which we aggregately refer to as optimization friendliness,
are identified and analyzed by a set of attention- and optimization-related
diagnosis tools. With these properties, the new representations show strong
fine-tuning performance. Specifically, the contrastive self-supervised learning
methods are made as competitive in fine-tuning as the state-of-the-art masked
image modeling (MIM) algorithms. The CLIP models' fine-tuning performance is
also significantly improved, with a CLIP ViT-L model reaching 89.0% top-1
accuracy on ImageNet-1K classification. On the 3-billion-parameter SwinV2-G
model, the fine-tuning accuracy is improved by +1.5 mIoU / +1.1 mAP to 61.4
mIoU / 64.2 mAP on ADE20K semantic segmentation and COCO object detection,
respectively, creating new records on both benchmarks. More importantly, our
work provides a way for the future research to focus more effort on the
generality and scalability of the learnt representations without being
pre-occupied with optimization friendliness since it can be enhanced rather
easily. The code will be available at
this https URL

研究发现通过简单的特征蒸馏，可以极大地提高预训练方法的微调性能，使其与基于遮挡图像模型的方法一样强大，在从事未来研究时，可以更加注重学习表示的普适性和可扩展性，而不需要过分关注其优化的友好性。