In this paper, we tackle an open research question in transfer learning,
which is selecting a model initialization to achieve high performance on a new
task, given several pre-trained models. We propose a new highly efficient and
accurate approach based on duality diagram similarity (DDS) between deep neural
networks (DNNs). DDS is a generic framework to represent and compare data of
different feature dimensions. We validate our approach on the Taskonomy dataset
by measuring the correspondence between actual transfer learning performance
rankings on 17 taskonomy tasks and predicted rankings. Computing DDS based
ranking for $17\times17$ transfers requires less than 2 minutes and shows a
high correlation ($0.86$) with actual transfer learning rankings, outperforming
state-of-the-art methods by a large margin ($10\%$) on the Taskonomy benchmark.
We also demonstrate the robustness of our model selection approach to a new
task, namely Pascal VOC semantic segmentation. Additionally, we show that our
method can be applied to select the best layer locations within a DNN for
transfer learning on 2D, 3D and semantic tasks on NYUv2 and Pascal VOC
datasets.

本文介绍了一种基于双重图相似度 (DDS) 的高效、准确的方法，可以在给定几个预训练模型时选择模型初始化，从而在新任务上实现高性能转移学习。我们在 Taskonomy 数据集上验证了我们的方法，并在 Pascal VOC 语义分割任务上展示了我们的模型选择方法的鲁棒性。