Few-shot learning allows pre-trained language models to adapt to downstream
tasks while using a limited number of training examples. However, practical
applications are limited when all model parameters must be optimized. In this
work we apply a new technique for parameter efficient few shot learning while
adopting a strict definition of parameter efficiency. Our training method
combines 1) intermediate training by reformulating natural language tasks as
entailment tasks \cite{wang_entailment_2021} and 2) differentiable optimization
of template and label tokens \cite{zhang_differentiable_2021}. We quantify the
tradeoff between parameter efficiency and performance in the few-shot regime
and propose a simple model agnostic approach that can be extended to any task
By achieving competitive performance while only optimizing 3\% of a model's
parameters and allowing for batched inference, we allow for more efficient
practical deployment of models.

本论文提出了一种新的技术，通过采用中间训练和可区分化优化的方式，在限制模型参数时实现 few-shot 学习的参数效率。作者通过实验量化了参数效率和性能在 few-shot 领域的权衡，并提出了一个简单的模型无关方法。

差分蕴涵用于参数高效的少样本学习

Differentiable Entailment for Parameter Efficient Few Shot Learning

Intermediate task fine-tuning has been shown to culminate in large transfer
gains across many NLP tasks. With an abundance of candidate datasets as well as
pre-trained language models, it has become infeasible to run the cross-product
of all combinations to find the best transfer setting. In this work we first
establish that similar sequential fine-tuning gains can be achieved in adapter
settings, and subsequently consolidate previously proposed methods that
efficiently identify beneficial tasks for intermediate transfer learning. We
experiment with a diverse set of 42 intermediate and 11 target English
classification, multiple choice, question answering, and sequence tagging
tasks. Our results show that efficient embedding based methods that rely solely
on the respective datasets outperform computational expensive few-shot
fine-tuning approaches. Our best methods achieve an average Regret@3 of less
than 1% across all target tasks, demonstrating that we are able to efficiently
identify the best datasets for intermediate training.

通过 adapter fine-tuning 方法，有效地和高效地获取并选取 NLP 任务中的中间任务，从而在目标任务上实现更好的效果。