Prompt engineering is a challenging and important task due to the high
sensitivity of Large Language Models (LLMs) to the given prompt and the
inherent ambiguity of a textual task instruction. Automatic prompt engineering
is essential to achieve optimized performance from LLMs. Recent studies have
demonstrated the capabilities of LLMs to automatically conduct prompt
engineering by employing a meta-prompt that incorporates the outcomes of the
last trials and proposes an improved prompt. However, this requires a
high-quality benchmark to compare different prompts, which is difficult and
expensive to acquire in many real-world use cases. In this work, we introduce a
new method for automatic prompt engineering, using a calibration process that
iteratively refines the prompt to the user intent. During the optimization
process, the system jointly generates synthetic data of boundary use cases and
optimizes the prompt according to the generated dataset. We demonstrate the
effectiveness of our method with respect to strong proprietary models on
real-world tasks such as moderation and generation. Our method outperforms
state-of-the-art methods with a limited number of annotated samples.
Furthermore, we validate the advantages of each one of the system's key
components. Our system is built in a modular way, facilitating easy adaptation
to other tasks. The code is available
$\href{this https URL}{here}$.

本研究介绍了一种自动提示工程的新方法，通过迭代完善用户意图的提示进程来优化系统性能，使用边界用例的合成数据进行优化，并验证了系统的关键组件优势。