Pre-trained large language models can efficiently interpolate human-written
prompts in a natural way. Multitask prompted learning can help generalization
through a diverse set of tasks at once, thus enhancing the potential for more
effective downstream fine-tuning. To perform efficient multitask-inference in
the same batch, parameter-efficient fine-tuning methods such as prompt tuning
have been proposed. However, the existing prompt tuning methods may lack
generalization. We propose SPT, a semi-parametric prompt tuning method for
multitask prompted learning. The novel component of SPT is a memory bank from
where memory prompts are retrieved based on discrete prompts. Extensive
experiments, such as (i) fine-tuning a full language model with SPT on 31
different tasks from 8 different domains and evaluating zero-shot
generalization on 9 heldout datasets under 5 NLP task categories and (ii)
pretraining SPT on the GLUE datasets and evaluating fine-tuning on the
SuperGLUE datasets, demonstrate effectiveness of SPT.

SPT 是一种半参数化的提示微调方法，其包含一个内存库，可以根据离散提示检索记忆提示，并通过 Fine-Tuning GLUE 数据集以及在五个自然语言处理任务类别下评估零 - shot 泛化以及在 SuperGLUE 数据集上预训练，众多实验证明了其有效性。