Training deep networks requires various design decisions regarding for
instance their architecture, data augmentation, or optimization. In this work,
we find these training variations to result in networks learning unique feature
sets from the data. Using public model libraries comprising thousands of models
trained on canonical datasets like ImageNet, we observe that for arbitrary
pairings of pretrained models, one model extracts significant data context
unavailable in the other -- independent of overall performance. Given any
arbitrary pairing of pretrained models and no external rankings (such as
separate test sets, e.g. due to data privacy), we investigate if it is possible
to transfer such "complementary" knowledge from one model to another without
performance degradation -- a task made particularly difficult as additional
knowledge can be contained in stronger, equiperformant or weaker models. Yet
facilitating robust transfer in scenarios agnostic to pretrained model pairings
would unlock auxiliary gains and knowledge fusion from any model repository
without restrictions on model and problem specifics - including from weaker,
lower-performance models. This work therefore provides an initial, in-depth
exploration on the viability of such general-purpose knowledge transfer. Across
large-scale experiments, we first reveal the shortcomings of standard knowledge
distillation techniques, and then propose a much more general extension through
data partitioning for successful transfer between nearly all pretrained models,
which we show can also be done unsupervised. Finally, we assess both the
scalability and impact of fundamental model properties on successful
model-agnostic knowledge transfer.

为了训练深度网络，需要在体系结构、数据增强或优化等方面作出各种设计决策。本文通过使用由数千个在 ImageNet 等经典数据集上训练的模型组成的公共模型库，在预训练模型的任意配对中发现了不同模型从数据中学习出独特的特征集。我们研究了在没有外部排名的情况下，是否能够在不降低性能的情况下从一种模型中转移这种 “互补” 知识到另一种模型，同时将强大、具有相似性能或较弱模型中的额外知识结合起来。通过大规模实验，我们揭示了标准知识蒸馏技术的缺点，并提出了一种更加通用的通过数据划分实现几乎所有预训练模型之间成功转移的方法，也证明了无监督转移的可能性。最后，我们评估了基本模型属性对成功的模型无关知识转移的可扩展性和影响。

惊人的收益及其发现之处：关于任何预训练模型之间普适知识转移存在和前景

Fantastic Gains and Where to Find Them: On the Existence and Prospect of  General Knowledge Transfer between Any Pretrained Model

We recently proposed a convolutional neural network (CNN) for remote sensing
image pansharpening obtaining a significant performance gain over the state of
the art. In this paper, we explore a number of architectural and training
variations to this baseline, achieving further performance gains with a
lightweight network which trains very fast. Leveraging on this latter property,
we propose a target-adaptive usage modality which ensures a very good
performance also in the presence of a mismatch w.r.t. the training set, and
even across different sensors. The proposed method, published online as an
off-the-shelf software tool, allows users to perform fast and high-quality
CNN-based pansharpening of their own target images on general-purpose hardware.

本文探究了基于卷积神经网络进行遥感图像全色增色的各种不同架构和训练变化，提出了一种目标自适应使用模式，并开发了一种可在通用硬件上快速进行高质量 CNN 基础全色增色的软件工具。