With the preponderance of pretrained deep learning models available
off-the-shelf from model banks today, finding the best weights to fine-tune to
your use-case can be a daunting task. Several methods have recently been
proposed to find good models for transfer learning, but they either don't scale
well to large model banks or don't perform well on the diversity of
off-the-shelf models. Ideally the question we want to answer is, "given some
data and a source model, can you quickly predict the model's accuracy after
fine-tuning?" In this paper, we formalize this setting as "Scalable Diverse
Model Selection" and propose several benchmarks for evaluating on this task. We
find that existing model selection and transferability estimation methods
perform poorly here and analyze why this is the case. We then introduce simple
techniques to improve the performance and speed of these algorithms. Finally,
we iterate on existing methods to create PARC, which outperforms all other
methods on diverse model selection. We have released the benchmarks and method
code in hope to inspire future work in model selection for accessible transfer
learning.

本文提出了一种基于 “可扩展多样化模型选择” 的框架，利用若干简单技巧改善了现有的预选权重和转移能力估计算法，在多种基于预训练深度学习模型的数据集上进行了广泛实验。结果表明，该方法（PARC）优于其他所有方法，并且本文公开了所用的 Benchmarks 和代码。