Masked Autoencoder~(MAE) is a prevailing self-supervised learning method that
achieves promising results in model pre-training. However, when the various
downstream tasks have data distributions different from the pre-training data,
the semantically irrelevant pre-training information might result in negative
transfer, impeding MAE's scalability. To address this issue, we propose a novel
MAE-based pre-training paradigm, Mixture of Cluster-conditional Experts (MoCE),
which can be trained once but provides customized pre-training models for
diverse downstream tasks. Different from the mixture of experts (MoE), our MoCE
trains each expert only with semantically relevant images by using
cluster-conditional gates. Thus, each downstream task can be allocated to its
customized model pre-trained with data most similar to the downstream data.
Experiments on a collection of 11 downstream tasks show that MoCE outperforms
the vanilla MAE by 2.45\% on average. It also obtains new state-of-the-art
self-supervised learning results on detection and segmentation.

提出了一种新的基于混合的聚类条件专家（MoCE）的 MAE 预训练范式，为不同的下游任务提供定制化的预训练模型，通过使用聚类条件门将每个专家仅与语义相关的图像进行训练，从而克服了 MAE 可扩展性中的负迁移问题。