Prompt Tuning (PT) has been largely successful as a parameter-efficient way of conditioning large-scale pre-trained language models towards a downstream task. More recently, soft prompt tuning has aimed to learn a fixed set of task-specific continuous vectors, i.e., soft tokens that remain static across the task samples. However, a fixed prompt may not generalize well to the diverse kinds of inputs the task comprises. With this motivation, we propose a novel way of prompting, Vector-quantized Input-contextualized Prompt Tuning or VIP. Essentially, VIP focuses on two aspects i) input-adaptation: input-specific contextualization of the soft tokens; and ii) vector quantization: we pass the tokens through a quantizer which effectively reduces representation variance by sampling prompts from a compact latent space. Over a wide range of natural language understanding tasks (SuperGLUE, QA, Relation Classification, NER, NLI), our proposed VIP framework beats the PT model by a margin of 1.19\%. Additionally, on Out-of-domain QA and Multi-Task setups over 4 different tasks spanning over 12 domains, we find that VIP outperforms PT by 0.75\%.

提出了 VIP，这是软提示调整框架的扩展，着重于两个方面 - 上下文提示通过小规模句子编码器学习软提示令牌的输入特定上下文化和量化提示通过向量量化网络将上下文提示映射到可学习的码本向量的设置。VIP 在各种语言理解任务上表现出优越性能，比软提示调整（PT）基线平均高出 1.19％，并且我们的一般化研究表明 VIP 学习了更稳健的提示表示。

自然语言理解的向量量化输入上下文化软提示