Transfer-learning methods aim to improve performance in a data-scarce target
domain using a model pretrained on a data-rich source domain. A cost-efficient
strategy, linear probing, involves freezing the source model and training a new
classification head for the target domain. This strategy is outperformed by a
more costly but state-of-the-art method -- fine-tuning all parameters of the
source model to the target domain -- possibly because fine-tuning allows the
model to leverage useful information from intermediate layers which is
otherwise discarded by the later pretrained layers. We explore the hypothesis
that these intermediate layers might be directly exploited. We propose a
method, Head-to-Toe probing (Head2Toe), that selects features from all layers
of the source model to train a classification head for the target-domain. In
evaluations on the VTAB-1k, Head2Toe matches performance obtained with
fine-tuning on average while reducing training and storage cost hundred folds
or more, but critically, for out-of-distribution transfer, Head2Toe outperforms
fine-tuning.

本文介绍了一种名为 Head-to-Toe probing（Head2Toe）的新型迁移学习方法，在 VTAB-1k 数据集上通过选择源模型所有层的特征进行目标领域的分类，能够在保证模型性能的情况下将训练和存储成本大幅度降低，并且对于分布外迁移，Head2Toe 方法优于微调。