The pretrain-finetune paradigm in modern computer vision facilitates the
success of self-supervised learning, which tends to achieve better
transferability than supervised learning. However, with the availability of
massive labeled data, a natural question emerges: how to train a better model
with both self and full supervision signals? In this paper, we propose
Omni-suPErvised Representation leArning with hierarchical supervisions (OPERA)
as a solution. We provide a unified perspective of supervisions from labeled
and unlabeled data and propose a unified framework of fully supervised and
self-supervised learning. We extract a set of hierarchical proxy
representations for each image and impose self and full supervisions on the
corresponding proxy representations. Extensive experiments on both
convolutional neural networks and vision transformers demonstrate the
superiority of OPERA in image classification, segmentation, and object
detection. Code is available at: this https URL.

该研究提出了一种名为 OPERA 的 Omni-suPErvised Representation leArning 方法，通过全面利用受监督和无监督的数据，将自监督学习和完全监督学习统一到一个框架中，以获得更好的模型性能。该方法通过使用层次化监督对每个图像提取一组代理表示，并在相应的代理表示上施加自监督和完全监督信号，展现了在图像分类、分割和目标检测上性能的稳定提升。

OPERA: 带有层级监督的全方位监督表示学习

OPERA: Omni-Supervised Representation Learning with Hierarchical Supervisions

The pretrain-finetune paradigm is a classical pipeline in visual learning.
Recent progress on unsupervised pretraining methods shows superior transfer
performance to their supervised counterparts. This paper revisits this
phenomenon and sheds new light on understanding the transferability gap between
unsupervised and supervised pretraining from a multilayer perceptron (MLP)
perspective. While previous works focus on the effectiveness of MLP on
unsupervised image classification where pretraining and evaluation are
conducted on the same dataset, we reveal that the MLP projector is also the key
factor to better transferability of unsupervised pretraining methods than
supervised pretraining methods. Based on this observation, we attempt to close
the transferability gap between supervised and unsupervised pretraining by
adding an MLP projector before the classifier in supervised pretraining. Our
analysis indicates that the MLP projector can help retain intra-class variation
of visual features, decrease the feature distribution distance between
pretraining and evaluation datasets, and reduce feature redundancy. Extensive
experiments on public benchmarks demonstrate that the added MLP projector
significantly boosts the transferability of supervised pretraining, e.g. +7.2%
top-1 accuracy on the concept generalization task, +5.8% top-1 accuracy for
linear evaluation on 12-domain classification tasks, and +0.8% AP on COCO
object detection task, making supervised pretraining comparable or even better
than unsupervised pretraining.

这篇论文研究了预训练模型下的无监督预训练与监督预训练的可迁移性差异，并提出了利用多层感知器解决可迁移性问题的方法，实验证明，在监督预训练模型中加入多层感知器可使模型的可迁移性显著提高，使得监督学习下的预训练能够与无监督学习相媲美或者更优秀。