Prompt-Tuning is a new paradigm for finetuning pre-trained language models in
a parameter-efficient way. Here, we explore the use of HyperNetworks to
generate hyper-prompts: we propose HyperPrompt, a novel architecture for
prompt-based task-conditioning of self-attention in Transformers. The
hyper-prompts are end-to-end learnable via generation by a HyperNetwork.
HyperPrompt allows the network to learn task-specific feature maps where the
hyper-prompts serve as task global memories for the queries to attend to, at
the same time enabling flexible information sharing among tasks. We show that
HyperPrompt is competitive against strong multi-task learning baselines with as
few as $0.14\%$ of additional task-conditioning parameters, achieving great
parameter and computational efficiency. Through extensive empirical
experiments, we demonstrate that HyperPrompt can achieve superior performances
over strong T5 multi-task learning baselines and parameter-efficient adapter
variants including Prompt-Tuning and HyperFormer++ on Natural Language
Understanding benchmarks of GLUE and SuperGLUE across many model sizes.

HyperPrompt 是一种基于 HyperNetwork 的结构，通过生成超级提示来为自注意力模型提供任务条件，使得在任务之间可以灵活地共享信息和实现参数和计算效率，从而在 GLUE 和 SuperGLUE 的自然语言理解基准测试中表现出优异的性能。