Parameter-efficient fine-tuning (PEFT) has shown its effectiveness in
adapting the pre-trained language models to downstream tasks while only
updating a small number of parameters. Despite the success, most existing
methods independently adapt to each task without considering knowledge transfer
between tasks and are limited to low-data regimes. To overcome this issue, we
propose Prototype-based HyperAdapter (PHA), a novel framework built on the
adapter-tuning and hypernetwork. It introduces an instance-dense retriever and
a prototypical hypernetwork to generate the conditional modules in a
sample-efficient manner. This leads to comparable performance improvements
against existing PEFT methods on multi-task learning and few-shot transfer
learning. More importantly, when the available data size gets smaller, our
method outperforms other strong baselines by a large margin. Based on our
extensive empirical experiments across various datasets, we demonstrate that
PHA strikes a better trade-off between trainable parameters, accuracy on stream
tasks, and sample efficiency.

基于 Protoype 的 HyperAdapter（PHA）是一个建立在 adapter-tuning 和超网络的新框架，它采用实例密集检索器和原型超网络以更有效地生成条件模块，从而在多任务学习和少样本迁移学习中与现有的参数有效微调（PEFT）方法相比具有可比较的性能改进，并在数据规模较小时，与其他强基准相比，取得了显著的优势。基于对各种数据集的大量实证实验，我们证明 PHA 在可训练参数、模型精度和样本效率之间取得了更好的平衡。