How do we transfer the relevant knowledge from ever larger foundation models
into small, task-specific downstream models that can run at much lower costs?
Standard transfer learning using pre-trained weights as the initialization
transfers limited information and commits us to often massive pre-trained
architectures. This procedure also precludes combining multiple pre-trained
models that learn complementary information. To address these shortcomings, we
introduce Adaptive Feature Transfer (AFT). Instead of transferring weights, AFT
operates purely on features, thereby decoupling the choice of the pre-trained
model from the smaller downstream model. Rather than indiscriminately
compressing all pre-trained features, AFT adaptively transfers pre-trained
features that are most useful for performing the downstream task, using a
simple regularization that adds minimal overhead. Across multiple vision,
language, and multi-modal datasets, AFT achieves significantly better
downstream performance compared to alternatives with a similar computational
cost. Furthermore, AFT reliably translates improvement in pre-trained models
into improvement in downstream performance, even if the downstream model is
over $50\times$ smaller, and can effectively transfer complementary information
learned by multiple pre-trained models.

在下游模型中以更低的成本运行的小型任务特定的下游模型中如何将相关知识从越来越大的基础模型转化过来。通过使用预训练权重作为初始化的标准迁移学习仅转移有限的信息，并且通常会导致庞大的预训练架构。为了解决这些缺点，我们引入自适应特征传输（AFT），它仅在特征上运行，从而将预训练模型的选择与较小的下游模型分离。AFT 自适应地传输对执行下游任务最有用的预训练特征，使用简单的正则化方法添加最小的开销。在多个视觉、语言和多模态数据集上，与类似的计算成本的替代方法相比，AFT 实现了显著更好的下游性能。此外，AFT 可靠地将预训练模型的改进转化为下游性能的提升，即使下游模型的规模缩小了 50 倍以上，也能有效地传输多个预训练模型学到的互补信息。