Prompt tuning is a parameter-efficient approach to adapting pre-trained
language models to downstream tasks. Although prompt tuning has been shown to
match the performance of full model tuning when training data is sufficient, it
tends to struggle in few-shot learning settings. In this paper, we present
Multi-task Pre-trained Modular Prompt (MP2) to boost prompt tuning for few-shot
learning. MP2 is a set of combinable prompts pre-trained on 38 Chinese tasks.
On downstream tasks, the pre-trained prompts are selectively activated and
combined, leading to strong compositional generalization to unseen tasks. To
bridge the gap between pre-training and fine-tuning, we formulate upstream and
downstream tasks into a unified machine reading comprehension task. Extensive
experiments under two learning paradigms, i.e., gradient descent and black-box
tuning, show that MP2 significantly outperforms prompt tuning, full model
tuning, and prior prompt pre-training methods in few-shot settings. In
addition, we demonstrate that MP2 can achieve surprisingly fast and strong
adaptation to downstream tasks by merely learning 8 parameters to combine the
pre-trained modular prompts.

本文提出了一种名为 Multi-task Pre-trained Modular Prompt (MP2) 的方法，通过预训练可组合的 prompts 并进行选择性的激活和组合，加强 Prompt Tuning 在 few-shot learning 的应用。通过 Black-box tuning 和 Gradient descent 两种学习方式的实验，表明 MP2 在 few-shot 设置下明显优于 Prompt Tuning、Full model Tuning 以及先前的 Prompt pre-training 方法。此外，我们还证明了 MP2 可以通过学习仅有 8 个参数来实现令人惊讶的快速适应 Downstream tasks。