An increasing number of well-trained deep networks have been released online
by researchers and developers, enabling the community to reuse them in a
plug-and-play way without accessing the training annotations. However, due to
the large number of network variants, such public-available trained models are
often of different architectures, each of which being tailored for a specific
task or dataset. In this paper, we study a deep-model reusing task, where we
are given as input pre-trained networks of heterogeneous architectures
specializing in distinct tasks, as teacher models. We aim to learn a
multitalented and light-weight student model that is able to grasp the
integrated knowledge from all such heterogeneous-structure teachers, again
without accessing any human annotation. To this end, we propose a common
feature learning scheme, in which the features of all teachers are transformed
into a common space and the student is enforced to imitate them all so as to
amalgamate the intact knowledge. We test the proposed approach on a list of
benchmarks and demonstrate that the learned student is able to achieve very
promising performance, superior to those of the teachers in their specialized
tasks.

本研究使用预先训练的网络作为示范，旨在学习一种能够整合各种异构结构教师的多才多艺、轻量级的学生模型，而无需获取任何人工注释信息。为此，提出了一种通用的特征学习方案，所有教师的特征都被转换成共同的空间，并且要求学生模型模仿他们的所有特征，以使完整的知识得以融合，并在一系列基准测试中证明所提出的方法的良好性能。