We describe PromptBoosting, a query-efficient procedure for building a text
classifier from a neural language model (LM) without access to the LM's
parameters, gradients, or hidden representations. This form of "black-box"
classifier training has become increasingly important as the cost of training
and inference in large-scale LMs grows. But existing black-box LM classifier
learning approaches are themselves computationally inefficient, typically
specializing LMs to the target task by searching in a large space of (discrete
or continuous) prompts using zeroth-order optimization methods. Instead of
directly optimizing in prompt space, PromptBoosting obtains a small pool of
prompts via a gradient-free approach and then constructs a large pool of weak
learners by pairing these prompts with different elements of the LM's output
distribution. These weak learners are then ensembled using the AdaBoost
algorithm. The entire learning process requires only a small number of forward
passes and no backward pass. Experiments show that PromptBoosting achieves
state-of-the-art performance in multiple black-box few-shot classification
tasks, and matches or outperforms full fine-tuning in both few-shot and
standard learning paradigms, while training 10x faster than existing black-box
methods.

本文提出了 PromptBoosting，一种由神经语言模型（LM）构建文本分类器的高效查询过程，其通过无梯度、无隐藏表示获取 LM 分类器的表现。该方法在黑盒分类器训练中越来越重要，并通过梯度自由方法获得一小组提示，并将其与 LM 输出分布的不同元素配对以建立大量弱学习器，然后使用 AdaBoost 算法将这些弱学习器整合。该学习方法仅需要进行少量前向传递和无反向传递的过程。实验表明，PromptBoosting 可在多个黑盒少样本分类任务中实现最先进的性能，并与完全微调方法在少样本和标准学习范式中达到或超过前者的表现，同时比现有黑盒方法训练速度快 10 倍。