Previous work has shown that there exists a scaling law between the size of
Language Models (LMs) and their zero-shot performance on different downstream
NLP tasks. In this work, we show that this phenomenon does not hold when
evaluating large LMs on tasks with negated prompts, but instead shows an
inverse scaling law. We evaluate 9 different tasks with negated prompts on (1)
pretrained LMs (OPT & GPT-3) of varying sizes (125M - 175B), (2) LMs further
pretrained to generalize to novel prompts (InstructGPT), (3) LMs provided with
few-shot examples, and (4) LMs fine-tuned specifically on negated prompts; all
LM types perform worse on negated prompts as they scale and show a huge
performance gap between the human performance when comparing the average score
on both original and negated prompts. By highlighting a critical limitation of
existing LMs and methods, we urge the community to develop new approaches of
developing LMs that actually follow the given instructions. We provide the code
and the datasets to explore negated prompts at
this https URL

本研究发现，在处理否定提示的任务时，语言模型的大小与其零 - shot 表现之间不存在按比例的关系，而是一种倒数的规律，并指出目前的 LMs 和方法存在重要限制。作者呼吁学术界开发新的方法来开发真正遵循给定指令的 LMs，并提供了探索否定提示的代码和数据集。