In this paper, we investigate a novel deep-model reusing task. Our goal is to
train a lightweight and versatile student model, without human-labelled
annotations, that amalgamates the knowledge and masters the expertise of two
pretrained teacher models working on heterogeneous problems, one on scene
parsing and the other on depth estimation. To this end, we propose an
innovative training strategy that learns the parameters of the student
intertwined with the teachers, achieved by 'projecting' its amalgamated
features onto each teacher's domain and computing the loss. We also introduce
two options to generalize the proposed training strategy to handle three or
more tasks simultaneously. The proposed scheme yields very encouraging results.
As demonstrated on several benchmarks, the trained student model achieves
results even superior to those of the teachers in their own expertise domains
and on par with the state-of-the-art fully supervised models relying on
human-labelled annotations.

本论文提出一种新的深度学习模型重用任务，利用预训练的教师模型在不需要人工标注的情况下训练一个轻量级和通用的学生模型，通过将合并特征投影到每个教师的领域并计算损失来学习学生模型。该方案在多项基准测试中实现了非常鼓舞人心的结果，甚至超过了教师模型在其自己的专业领域中的表现。