By conditioning on natural language instructions, large language models
(LLMs) have displayed impressive capabilities as general-purpose computers.
However, task performance depends significantly on the quality of the prompt
used to steer the model, and most effective prompts have been handcrafted by
humans. Inspired by classical program synthesis and the human approach to
prompt engineering, we propose Automatic Prompt Engineer (APE) for automatic
instruction generation and selection. In our method, we treat the instruction
as the "program," optimized by searching over a pool of instruction candidates
proposed by an LLM in order to maximize a chosen score function. To evaluate
the quality of the selected instruction, we evaluate the zero-shot performance
of another LLM following the selected instruction. Experiments on 24 NLP tasks
show that our automatically generated instructions outperform the prior LLM
baseline by a large margin and achieve better or comparable performance to the
instructions generated by human annotators on 19/24 tasks. We conduct extensive
qualitative and quantitative analyses to explore the performance of APE. We
show that APE-engineered prompts can be applied to steer models toward
truthfulness and/or informativeness, as well as to improve few-shot learning
performance by simply prepending them to standard in-context learning prompts.
Please check out our webpage at
this https URL.

使用语言模型作自然语言指导的条件，我们提出了自动提示工程师（APE），通过搜索竞选一组精心设计的自动生成的提示中的最佳提示来优化指令来解决 NLP 任务中的性能问题，并实现对模型的更好的指导和性能提升。在 24 个任务中，通过自动产生的新指令，我们的 APE 方法可使性能提高，并在 19 个任务中的性能甚至好于人类注释者生成的指令。通过大量定性和定量分析，我们证明了 APE 的有效性。