Consistently scaling pre-trained language models (PLMs) imposes substantial
burdens on model adaptation, necessitating more efficient alternatives to
conventional fine-tuning. Given the advantage of prompting in the zero-shot
setting and the observed performance fluctuation among different prompts, we
explore the instance-level prompt and their generalizability. By searching
through the prompt space, we first validate the assumption that for every
instance, there is almost always a lottery prompt that induces the correct
prediction from the PLM, and such prompt can be obtained at a low cost thanks
to the inherent ability of PLMs. Meanwhile, we find that some strong lottery
prompts have high performance over the whole training set, and they are
equipped with distinguishable linguistic features. Lastly, we attempt to
generalize the searched strong lottery prompts to unseen data with prompt
ensembling method without any parameter tuning. Experiments are conducted on
various types of NLP classification tasks and demonstrate that the proposed
method can achieve comparable results with other gradient-free and
optimization-free baselines.

本文提出了一种基于逐个样例级别的 Prompt 搜索方法，然后使用 Prompt 集成方法将搜索到的强彩票 Prompts 推广到未见过的数据上。实验结果表明，该方法与其他无梯度、无优化基线方法相比能够取得可比的结果。

探究预训练语言模型的抽奖提示

Exploring Lottery Prompts for Pre-trained Language Models

Self-supervised pre-training of transformer models has shown enormous success
in improving performance on a number of downstream tasks. However, fine-tuning
on a new task still requires large amounts of task-specific labelled data to
achieve good performance. We consider this problem of learning to generalize to
new tasks with few examples as a meta-learning problem. While meta-learning has
shown tremendous progress in recent years, its application is still limited to
simulated problems or problems with limited diversity across tasks. We develop
a novel method, LEOPARD, which enables optimization-based meta-learning across
tasks with different number of classes, and evaluate different methods on
generalization to diverse NLP classification tasks. LEOPARD is trained with the
state-of-the-art transformer architecture and shows better generalization to
tasks not seen at all during training, with as few as 4 examples per label.
Across 17 NLP tasks, including diverse domains of entity typing, natural
language inference, sentiment analysis, and several other text classification
tasks, we show that LEOPARD learns better initial parameters for few-shot
learning than self-supervised pre-training or multi-task training,
outperforming many strong baselines, for example, yielding 14.5% average
relative gain in accuracy on unseen tasks with only 4 examples per label.

LEOPARD 是一种基于元学习的方法，使用少量标记的数据可优化跨任务进行学习，适用于 NLP 分类任务中的不同类数，并表现出比自监督预训练或多任务训练更好的泛化能力。