Conventional wisdom suggests parameter-efficient fine-tuning of foundation
models as the state-of-the-art method for transfer learning in vision,
replacing the rich literature of alternatives such as meta-learning. In trying
to harness the best of both worlds, meta-tuning introduces a subsequent
optimization stage of foundation models but has so far only shown limited
success and crucially tends to underperform on out-of-domain (OOD) tasks. In
this paper, we introduce Sparse MetA-Tuning (SMAT), a method inspired by sparse
mixture-of-experts approaches and trained to isolate subsets of pre-trained
parameters automatically for meta-tuning on each task. SMAT successfully
overcomes OOD sensitivity and delivers on the promise of enhancing the transfer
abilities of vision foundation models beyond parameter-efficient finetuning. We
establish new state-of-the-art results on a challenging combination of
Meta-Dataset augmented with additional OOD tasks in both zero-shot and
gradient-based adaptation settings. In addition, we provide a thorough analysis
of the superiority of learned over hand-designed sparsity patterns for sparse
expert methods and the pivotal importance of the sparsity level in balancing
between in-domain and out-of-domain generalization. Our code is publicly
available.

通过稀疏化混合专家方法，稀疏元调优成功地提高了视觉基础模型的迁移能力，并在零阶和基于梯度的适应环境中建立了新的最先进的结果。