Recently, a boom of papers has shown extraordinary progress in zero-shot and
few-shot learning with various prompt-based models. It is commonly argued that
prompts help models to learn faster in the same way that humans learn faster
when provided with task instructions expressed in natural language. In this
study, we experiment with over 30 prompt templates manually written for natural
language inference (NLI). We find that models learn just as fast with many
prompts that are intentionally irrelevant or even pathologically misleading as
they do with instructively "good" prompts. Further, such patterns hold even for
models as large as 175 billion parameters (Brown et al., 2020) as well as the
recently proposed instruction-tuned models which are trained on hundreds of
prompts (Sanh et al., 2022). That is, instruction-tuned models often produce
good predictions with irrelevant and misleading prompts even at zero shots. In
sum, notwithstanding prompt-based models' impressive improvement, we find
evidence of serious limitations that question the degree to which such
improvement is derived from models understanding task instructions in ways
analogous to humans' use of task instructions.

本研究证实了即使是故意无关或甚至有误导性的提示，也可以让一些零样本或少样本学习模型像 “好” 的提示一样快地进行学习，这一模式适用于模型尺寸大小，这引出一个问题，问这种改进是不是源于模型像人类一样理解掌握任务提示。