We propose a new strategy for applying large pre-trained language models to
novel tasks when labeled training data is limited. Rather than apply the model
in a typical zero-shot or few-shot fashion, we treat the model as the basis for
labeling functions in a weak supervision framework. To create a classifier, we
first prompt the model to answer multiple distinct queries about an example and
define how the possible responses should be mapped to votes for labels and
abstentions. We then denoise these noisy label sources using the Snorkel system
and train an end classifier with the resulting training data. Our experimental
evaluation shows that prompting large language models within a weak supervision
framework can provide significant gains in accuracy. On the WRENCH weak
supervision benchmark, this approach can significantly improve over zero-shot
performance, an average 19.5% reduction in errors. We also find that this
approach produces classifiers with comparable or superior accuracy to those
trained from hand-engineered rules.

本研究提出使用大型预训练语言模型进行弱监督学习的策略，并使用 Snorkel 系统去噪声标签，得到的训练数据可提高分类器的准确性，相比于零样本方法，错误率平均降低 19.5%。此外，该方法所得分类器的准确度相当或高于手动设置的规则。