As large language models (LLMs) are adopted as a fundamental component of
language technologies, it is crucial to accurately characterize their
performance. Because choices in prompt design can strongly influence model
behavior, this design process is critical in effectively using any modern
pre-trained generative language model. In this work, we focus on LLM
sensitivity to a quintessential class of meaning-preserving design choices:
prompt formatting. We find that several widely used open-source LLMs are
extremely sensitive to subtle changes in prompt formatting in few-shot
settings, with performance differences of up to 76 accuracy points when
evaluated using LLaMA-2-13B. Sensitivity remains even when increasing model
size, the number of few-shot examples, or performing instruction tuning. Our
analysis suggests that work evaluating LLMs with prompting-based methods would
benefit from reporting a range of performance across plausible prompt formats,
instead of the currently-standard practice of reporting performance on a single
format. We also show that format performance only weakly correlates between
models, which puts into question the methodological validity of comparing
models with an arbitrarily chosen, fixed prompt format. To facilitate
systematic analysis we propose FormatSpread, an algorithm that rapidly
evaluates a sampled set of plausible prompt formats for a given task, and
reports the interval of expected performance without accessing model weights.
Furthermore, we present a suite of analyses that characterize the nature of
this sensitivity, including exploring the influence of particular atomic
perturbations and the internal representation of particular formats.

大语言模型的关键因素在于提示设计对其性能会产生巨大影响，通过许多方面的分析研究，我们发现几种常用开源大语言模型对微妙的提示格式变化具有极高的敏感性，这种敏感性在不断增加模型大小、样本数量或进行指令调整时仍然存在。