Conventional wisdom suggests parameter-efficient fine-tuning of foundation
models as the state-of-the-art method for transfer learning in vision,
replacing the rich literature of alternatives such as meta-learning. In trying
to harness the best of both worlds, meta-tuning introduces a subsequent
optimization stage of foundation models but has so far only shown limited
success and crucially tends to underperform on out-of-domain (OOD) tasks. In
this paper, we introduce Sparse MetA-Tuning (SMAT), a method inspired by sparse
mixture-of-experts approaches and trained to isolate subsets of pre-trained
parameters automatically for meta-tuning on each task. SMAT successfully
overcomes OOD sensitivity and delivers on the promise of enhancing the transfer
abilities of vision foundation models beyond parameter-efficient finetuning. We
establish new state-of-the-art results on a challenging combination of
Meta-Dataset augmented with additional OOD tasks in both zero-shot and
gradient-based adaptation settings. In addition, we provide a thorough analysis
of the superiority of learned over hand-designed sparsity patterns for sparse
expert methods and the pivotal importance of the sparsity level in balancing
between in-domain and out-of-domain generalization. Our code is publicly
available.

通过稀疏化混合专家方法，稀疏元调优成功地提高了视觉基础模型的迁移能力，并在零阶和基于梯度的适应环境中建立了新的最先进的结果。

通过稀疏插值专家释放元调优的强大力量，以实现少样本泛化

Unleashing the Power of Meta-tuning for Few-shot Generalization Through  Sparse Interpolated Experts

Large language models (LLMs) have shown impressive performance in following
natural language instructions to solve unseen tasks. However, it remains
unclear whether models truly understand task definitions and whether the
human-written definitions are optimal. In this paper, we systematically study
the role of task definitions in instruction learning. We first conduct an
ablation analysis informed by human annotations to understand which parts of a
task definition are most important, and find that model performance only drops
substantially when removing contents describing the task output, in particular
label information. Next, we propose an automatic algorithm to compress task
definitions to a minimal supporting set of tokens, and find that 60\% of tokens
can be removed while maintaining or even improving model performance. Based on
these results, we propose two strategies to help models better leverage task
instructions: (1) providing only key information for tasks in a common
structured format, and (2) adding a meta-tuning stage to help the model better
understand the definitions. With these two strategies, we achieve a 4.2 Rouge-L
improvement over 119 unseen test tasks.

该研究使用人工注释的消融分析方法研究任务定义在指令学习中的作用，并提出两种策略以帮助大型语言模型更好地利用任务指令，包括提供以常见结构化格式呈现关键信息的任务，并添加一个元调整阶段，以帮助模型更好地理解定义。应用这两种策略，在 119 个未见过的测试任务中取得了 4.2 Rouge-L 的提高。

读了说明书吗？重新思考指导学习中任务定义的有效性

Did You Read the Instructions? Rethinking the Effectiveness of Task  Definitions in Instruction Learning

Large pre-trained language models (LMs) such as GPT-3 have acquired a
surprising ability to perform zero-shot learning. For example, to classify
sentiment without any training examples, we can "prompt" the LM with the review
and the label description "Does the user like this movie?", and ask whether the
next word is "yes" or "no". However, the next word prediction training
objective is still misaligned with the target zero-shot learning objective. To
address this weakness, we propose meta-tuning, which directly optimizes the
zero-shot learning objective by fine-tuning pre-trained language models on a
collection of datasets. We focus on classification tasks, and construct the
meta-dataset by aggregating 43 existing datasets and annotating 441 label
descriptions in a question-answering (QA) format. When evaluated on unseen
tasks, meta-tuned models outperform a same-sized QA model and the previous SOTA
zero-shot learning system based on natural language inference. Additionally,
increasing parameter count from 220M to 770M improves AUC-ROC scores by 6.3%,
and we forecast that even larger models would perform better. Therefore,
measuring zero-shot learning performance on language models out-of-the-box
might underestimate their true potential, and community-wide efforts on
aggregating datasets and unifying their formats can help build models that
answer prompts better.

该论文提出了元调谐（meta-tuning）方法，旨在直接针对零样本学习目标对预训练语言模型进行微调，应用于分类任务，并通过聚合 43 个现有的数据集和 441 个标签描述来构建元数据集。实验证明，相比基于自然语言推理的先前 SOTA 零样本学习系统以及同样大小的 QA 模型，元调整模型在新的任务上表现更好，同时我们认为，增加参数数量会进一步提升 AUC-ROC 分数。