The latest generation of LLMs can be prompted to achieve impressive zero-shot
or few-shot performance in many NLP tasks. However, since performance is highly
sensitive to the choice of prompts, considerable effort has been devoted to
crowd-sourcing prompts or designing methods for prompt optimisation. Yet, we
still lack a systematic understanding of how linguistic properties of prompts
correlate with task performance. In this work, we investigate how LLMs of
different sizes, pre-trained and instruction-tuned, perform on prompts that are
semantically equivalent, but vary in linguistic structure. We investigate both
grammatical properties such as mood, tense, aspect and modality, as well as
lexico-semantic variation through the use of synonyms. Our findings contradict
the common assumption that LLMs achieve optimal performance on lower perplexity
prompts that reflect language use in pretraining or instruction-tuning data.
Prompts transfer poorly between datasets or models, and performance cannot
generally be explained by perplexity, word frequency, ambiguity or prompt
length. Based on our results, we put forward a proposal for a more robust and
comprehensive evaluation standard for prompting research.

在这项研究中，我们调查了不同规模、预训练和指导调优的 LLMs 在在语义上等效但语言结构不同的提示上的性能，发现 LLMs 的性能不能通过困惑度、词频、歧义或提示长度来解释，并提出了对提示研究更健壮和综合的评估标准的建议。