Large Language Models (LLMs) have significantly enhanced Information
Retrieval (IR) across various modules, such as reranking. Despite impressive
performance, current zero-shot relevance ranking with LLMs heavily relies on
human prompt engineering. Existing automatic prompt engineering algorithms
primarily focus on language modeling and classification tasks, leaving the
domain of IR, particularly reranking, underexplored. Directly applying current
prompt engineering algorithms to relevance ranking is challenging due to the
integration of query and long passage pairs in the input, where the ranking
complexity surpasses classification tasks. To reduce human effort and unlock
the potential of prompt optimization in reranking, we introduce a novel
automatic prompt engineering algorithm named APEER. APEER iteratively generates
refined prompts through feedback and preference optimization. Extensive
experiments with four LLMs and ten datasets demonstrate the substantial
performance improvement of APEER over existing state-of-the-art (SoTA) manual
prompts. Furthermore, we find that the prompts generated by APEER exhibit
better transferability across diverse tasks and LLMs. Code is available at
this https URL

通过自动提示工程算法 APEER，大型语言模型在信息检索中的零样本相关性排名取得了显著的性能提升，并展现出更好的跨任务和跨语言模型的可迁移性。

APEER：自动提示工程提升大型语言模型重排

APEER: Automatic Prompt Engineering Enhances Large Language Model  Reranking

Prompt engineering is a challenging and important task due to the high
sensitivity of Large Language Models (LLMs) to the given prompt and the
inherent ambiguity of a textual task instruction. Automatic prompt engineering
is essential to achieve optimized performance from LLMs. Recent studies have
demonstrated the capabilities of LLMs to automatically conduct prompt
engineering by employing a meta-prompt that incorporates the outcomes of the
last trials and proposes an improved prompt. However, this requires a
high-quality benchmark to compare different prompts, which is difficult and
expensive to acquire in many real-world use cases. In this work, we introduce a
new method for automatic prompt engineering, using a calibration process that
iteratively refines the prompt to the user intent. During the optimization
process, the system jointly generates synthetic data of boundary use cases and
optimizes the prompt according to the generated dataset. We demonstrate the
effectiveness of our method with respect to strong proprietary models on
real-world tasks such as moderation and generation. Our method outperforms
state-of-the-art methods with a limited number of annotated samples.
Furthermore, we validate the advantages of each one of the system's key
components. Our system is built in a modular way, facilitating easy adaptation
to other tasks. The code is available
$\href{this https URL}{here}$.

本研究介绍了一种自动提示工程的新方法，通过迭代完善用户意图的提示进程来优化系统性能，使用边界用例的合成数据进行优化，并验证了系统的关键组件优势。

基于意图的提示校准：通过合成边界案例增强提示优化

Intent-based Prompt Calibration: Enhancing prompt optimization with  synthetic boundary cases

Prompt engineering is a challenging yet crucial task for optimizing the
performance of large language models (LLMs). It requires complex reasoning to
examine the model's errors, hypothesize what is missing or misleading in the
current prompt, and communicate the task with clarity. While recent works
indicate that LLMs can be meta-prompted to perform automatic prompt
engineering, their potentials may not be fully untapped due to the lack of
sufficient guidance to elicit complex reasoning capabilities in LLMs in the
meta-prompt. In this work, we investigate the problem of "prompt engineering a
prompt engineer" -- constructing a meta-prompt that more effectively guides
LLMs to perform automatic prompt engineering. We introduce and analyze key
components, such as a step-by-step reasoning template and context
specification, which lead to improved performance. In addition, inspired by
common optimization concepts such as batch size, step size and momentum, we
introduce their verbalized counterparts to the meta-prompt and investigate
their effects. Our final method, named PE2, finds a prompt that outperforms
"let's think step by step" by 6.3% on the MultiArith dataset and 3.1% on the
GSM8K dataset. To demonstrate its versatility, we apply PE2 to the Instruction
Induction benchmark, a suite of counterfactual tasks, and a lengthy, real-world
industrial prompt. In these settings, PE2 achieves strong performance and
outperforms prior automatic prompt engineering baselines. Further, we show that
PE2 makes meaningful and targeted prompt edits, amends erroneous or incomplete
prompts, and presents non-trivial counterfactual reasoning abilities.

通过构建元提示（meta-prompt）从而更有效地引导大型语言模型（LLMs）进行自动提示工程，进而优化其性能。

提示工程师的提示工程

Prompt Engineering a Prompt Engineer

By conditioning on natural language instructions, large language models
(LLMs) have displayed impressive capabilities as general-purpose computers.
However, task performance depends significantly on the quality of the prompt
used to steer the model, and most effective prompts have been handcrafted by
humans. Inspired by classical program synthesis and the human approach to
prompt engineering, we propose Automatic Prompt Engineer (APE) for automatic
instruction generation and selection. In our method, we treat the instruction
as the "program," optimized by searching over a pool of instruction candidates
proposed by an LLM in order to maximize a chosen score function. To evaluate
the quality of the selected instruction, we evaluate the zero-shot performance
of another LLM following the selected instruction. Experiments on 24 NLP tasks
show that our automatically generated instructions outperform the prior LLM
baseline by a large margin and achieve better or comparable performance to the
instructions generated by human annotators on 19/24 tasks. We conduct extensive
qualitative and quantitative analyses to explore the performance of APE. We
show that APE-engineered prompts can be applied to steer models toward
truthfulness and/or informativeness, as well as to improve few-shot learning
performance by simply prepending them to standard in-context learning prompts.
Please check out our webpage at
this https URL.

使用语言模型作自然语言指导的条件，我们提出了自动提示工程师（APE），通过搜索竞选一组精心设计的自动生成的提示中的最佳提示来优化指令来解决 NLP 任务中的性能问题，并实现对模型的更好的指导和性能提升。在 24 个任务中，通过自动产生的新指令，我们的 APE 方法可使性能提高，并在 19 个任务中的性能甚至好于人类注释者生成的指令。通过大量定性和定量分析，我们证明了 APE 的有效性。