As a fundamental problem in transfer learning, model selection aims to rank
off-the-shelf pre-trained models and select the most suitable one for the new
target task. Existing model selection techniques are often constrained in their
scope and tend to overlook the nuanced relationships between models and tasks.
In this paper, we present a pragmatic framework \textbf{Fennec}, delving into a
diverse, large-scale model repository while meticulously considering the
intricate connections between tasks and models. The key insight is to map all
models and historical tasks into a transfer-related subspace, where the
distance between model vectors and task vectors represents the magnitude of
transferability. A large vision model, as a proxy, infers a new task's
representation in the transfer space, thereby circumventing the computational
burden of extensive forward passes. We also investigate the impact of the
inherent inductive bias of models on transfer results and propose a novel
method called \textbf{archi2vec} to encode the intricate structures of models.
The transfer score is computed through straightforward vector arithmetic with a
time complexity of $\mathcal{O}(1)$. Finally, we make a substantial
contribution to the field by releasing a comprehensive benchmark. We validate
the effectiveness of our framework through rigorous testing on two benchmarks.
The benchmark and the code will be publicly available in the near future.

本文提出了一个实用性框架 Fennec，通过将所有模型和历史任务映射到一个 transfer-related 子空间中，以判断他们之间的可迁移性，并通过一个大型视觉模型来推断新任务在 transfer 空间中的表示，最后通过 rigorous testing 在两个基准测试上验证了框架的有效性。