Transferability metrics is a maturing field with increasing interest, which aims at providing heuristics for selecting the most suitable source models to transfer to a given target dataset, without fine-tuning them all. However, existing works rely on custom experimental setups which differ across papers, leading to inconsistent conclusions about which transferability metrics work best. In this paper we conduct a large-scale study by systematically constructing a broad range of 715k experimental setup variations. We discover that even small variations to an experimental setup lead to different conclusions about the superiority of a transferability metric over another. Then we propose better evaluations by aggregating across many experiments, enabling to reach more stable conclusions. As a result, we reveal the superiority of LogME at selecting good source datasets to transfer from in a semantic segmentation scenario, NLEEP at selecting good source architectures in an image classification scenario, and GBC at determining which target task benefits most from a given source model. Yet, no single transferability metric works best in all scenarios.

通过大规模实验研究，聚合不同实验结果并提出 LogME 转移度量法，发现在语义分割场景中选择良好的源数据集、在图像分类场景中选择良好的源体系结构以及确定哪个目标任务最受益于给定的源模型的最佳转移度量法是 LogME、NLEEP 和 GBC，但没有一种单一的转移度量法能够在所有场景中表现最佳。

转移度量评估的稳定性如何？