Derivative-free prompt learning has emerged as a lightweight alternative to
prompt tuning, which only requires model inference to optimize the prompts.
However, existing work did not take full advantage of the over-parameterized
characteristics of large pre-trained language models (PLMs). In this paper, we
propose Clip-Tuning, a simple yet effective method that adopts diverse frozen
"thinned" networks of PLMs to obtain a mixture of rewards and thus advance the
derivative-free prompt learning. The thinned networks consist of all the hidden
units that survive a stationary dropout strategy, whose inference predictions
reflect an ensemble of partial views over prompted training samples. Our method
outperforms previous gradient-free prompt learning methods and achieves parity
with gradient-based counterparts on seven language understanding benchmarks
under few-shot settings.

本文提出了一种名为 Clip-Tuning 的方法，利用预训练的语言模型来获得奖励的混合，并在少样本情况下超越了以前的基于梯度的对照方法。该方法采用了不同的冻结的预训练语言模型的精简网络，其推断预测反映了对提示培训样本的部分视图的集合。