The pretrain-finetune paradigm has shown outstanding performance on many
applications of deep learning, where a model is pre-trained on a upstream large
dataset (e.g. ImageNet), and is then fine-tuned to different downstream tasks.
Though for most cases, the pre-training stage is conducted based on supervised
methods, recent works on self-supervised pre-training have shown powerful
transferability and even outperform supervised pre-training on multiple
downstream tasks. It thus remains an open question how to better generalize
supervised pre-training model to downstream tasks. In this paper, we argue that
the worse transferability of existing supervised pre-training methods arise
from the negligence of valuable intra-class semantic difference. This is
because these methods tend to push images from the same class close to each
other despite of the large diversity in their visual contents, a problem to
which referred as "overfit of upstream tasks". To alleviate this problem, we
propose a new supervised pre-training method based on Leave-One-Out
K-Nearest-Neighbor, or LOOK for short. It relieves the problem of overfitting
upstream tasks by only requiring each image to share its class label with most
of its k nearest neighbors, thus allowing each class to exhibit a multi-mode
distribution and consequentially preserving part of intra-class difference for
better transferring to downstream tasks. We developed efficient implementation
of the proposed method that scales well to large datasets. Experimental studies
on multiple downstream tasks show that LOOK outperforms other state-of-the-art
methods for supervised and self-supervised pre-training.

该论文提出了一种名为 LOOK 的基于留一法 K 最近邻（Leave-One-Out K-Nearest-Neighbor）的监督预训练方法，它通过让每个图像只与其 k 个最近邻居共享其类标签，从而允许每个类展现出多模式分布，从而更好地转移至下游任务，并在多个下游任务上的实验研究表明，LOOK 优于其他监督和自监督预训练方法。