Training deep networks requires various design decisions regarding for
instance their architecture, data augmentation, or optimization. In this work,
we find these training variations to result in networks learning unique feature
sets from the data. Using public model libraries comprising thousands of models
trained on canonical datasets like ImageNet, we observe that for arbitrary
pairings of pretrained models, one model extracts significant data context
unavailable in the other -- independent of overall performance. Given any
arbitrary pairing of pretrained models and no external rankings (such as
separate test sets, e.g. due to data privacy), we investigate if it is possible
to transfer such "complementary" knowledge from one model to another without
performance degradation -- a task made particularly difficult as additional
knowledge can be contained in stronger, equiperformant or weaker models. Yet
facilitating robust transfer in scenarios agnostic to pretrained model pairings
would unlock auxiliary gains and knowledge fusion from any model repository
without restrictions on model and problem specifics - including from weaker,
lower-performance models. This work therefore provides an initial, in-depth
exploration on the viability of such general-purpose knowledge transfer. Across
large-scale experiments, we first reveal the shortcomings of standard knowledge
distillation techniques, and then propose a much more general extension through
data partitioning for successful transfer between nearly all pretrained models,
which we show can also be done unsupervised. Finally, we assess both the
scalability and impact of fundamental model properties on successful
model-agnostic knowledge transfer.

为了训练深度网络，需要在体系结构、数据增强或优化等方面作出各种设计决策。本文通过使用由数千个在 ImageNet 等经典数据集上训练的模型组成的公共模型库，在预训练模型的任意配对中发现了不同模型从数据中学习出独特的特征集。我们研究了在没有外部排名的情况下，是否能够在不降低性能的情况下从一种模型中转移这种 “互补” 知识到另一种模型，同时将强大、具有相似性能或较弱模型中的额外知识结合起来。通过大规模实验，我们揭示了标准知识蒸馏技术的缺点，并提出了一种更加通用的通过数据划分实现几乎所有预训练模型之间成功转移的方法，也证明了无监督转移的可能性。最后，我们评估了基本模型属性对成功的模型无关知识转移的可扩展性和影响。