Pre-training convolutional neural networks with weakly-supervised and self-supervised strategies is becoming increasingly popular for several computer vision tasks. However, due to the lack of strong discriminative signals, these learned representations may overfit to the pre-training objective (e.g., hashtag prediction) and not generalize well to downstream tasks. In this work, we present a simple strategy - ClusterFit (CF) to improve the robustness of the visual representations learned during pre-training. Given a dataset, we (a) cluster its features extracted from a pre-trained network using k-means and (b) re-train a new network from scratch on this dataset using cluster assignments as pseudo-labels. We empirically show that clustering helps reduce the pre-training task-specific information from the extracted features thereby minimizing overfitting to the same. Our approach is extensible to different pre-training frameworks -- weak- and self-supervised, modalities -- images and videos, and pre-training tasks -- object and action classification. Through extensive transfer learning experiments on 11 different target datasets of varied vocabularies and granularities, we show that ClusterFit significantly improves the representation quality compared to the state-of-the-art large-scale (millions / billions) weakly-supervised image and video models and self-supervised image models.

本文提出一种基于ClusterFit的简单策略来提高预训练期间学到的视觉表示的鲁棒性，通过对从预训练网络中提取的特征进行聚类，从而减少了特定于任务的信息从而将过度拟合降到最低。经过在11个不同目标数据集上的传递学习实验证明，与最先进的大规模（数百万/十亿）弱监督图像和视频模型和自监督图像模型相比，ClusterFit极大地提高了表示质量。

ClusterFit：改善视觉表示的泛化能力