Recent works have shown that large language models (LLMs) work remarkably
well on a wide range of tasks through in-context learning and optimization of
in-context examples (ICE). However, most of these studies assume either a fixed
or no instruction provided in the prompt, leading to the apparent consensus
that the optimization of in-context examples is critical for better
performance. We challenge this consensus for instruction-tuned LLMs by
investigating the necessity of optimizing in-context examples when
task-specific instructions are provided, and find that there are tasks for
which various ways of optimizing in-context examples yield diminishing returns.
We introduce a task-specific metric called \metriclong{} (\metric) that
quantifies the learnability of tasks from a given instruction, and provides a
heuristic that helps decide whether to optimize for instructions or ICE for any
new task. On a wide range of tasks and a systematically created instruction set
with gradually added details, we validate our hypothesis empirically by
computing \metric with query-dependent bins of examples, comparing different
instructions with ICE selection methods, and performing label perturbation
experiments. We conclude that tasks can be divided into two broad classes based
on the \metric metric, where the returns on ICE optimization follow predictable
trends when instructions are provided in the prompt.

大型语言模型通过上下文学习与优化可以在广泛的任务上取得显著效果，然而，该研究调查了当提供任务特定指令时，优化上下文示例的必要性，并发现对于某些任务，对上下文示例进行优化产生递减收益，为此引入了用于测量任务可学习性的度量方法，从而提供了一种启发式方法，帮助决定针对任何新任务是优化指令还是优化上下文示例。

优化上下文示例：是否为 NICE 的？

NICE: To Optimize In-Context Examples or Not?

Prompt engineering has emerged as an indispensable technique for extending
the capabilities of large language models (LLMs) and vision-language models
(VLMs). This approach leverages task-specific instructions, known as prompts,
to enhance model efficacy without modifying the core model parameters. Rather
than updating the model parameters, prompts allow seamless integration of
pre-trained models into downstream tasks by eliciting desired model behaviors
solely based on the given prompt. Prompts can be natural language instructions
that provide context to guide the model or learned vector representations that
activate relevant knowledge. This burgeoning field has enabled success across
various applications, from question-answering to commonsense reasoning.
However, there remains a lack of systematic organization and understanding of
the diverse prompt engineering methods and techniques. This survey paper
addresses the gap by providing a structured overview of recent advancements in
prompt engineering, categorized by application area. For each prompting
approach, we provide a summary detailing the prompting methodology, its
applications, the models involved, and the datasets utilized. We also delve
into the strengths and limitations of each approach and include a taxonomy
diagram and table summarizing datasets, models, and critical points of each
prompting technique. This systematic analysis enables a better understanding of
this rapidly developing field and facilitates future research by illuminating
open challenges and opportunities for prompt engineering.

本综述论文通过提供近期进展的结构化概述，对提示工程的不同方法和技术进行分类，详细介绍了提示方法学、应用领域、所使用的模型和数据集，并探讨了每种方法的优点、局限性以及通过分类图和表格总结了数据集、模型和关键点，从而更好地理解这一快速发展领域并为提示工程的未来研究提供洞察、揭示了开放性挑战和机遇。

大型语言模型中关于提示工程的系统调查：技术与应用

A Systematic Survey of Prompt Engineering in Large Language Models:  Techniques and Applications

We introduce meta-prompting, an effective scaffolding technique designed to
enhance the functionality of language models (LMs). This approach transforms a
single LM into a multi-faceted conductor, adept at managing and integrating
multiple independent LM queries. By employing high-level instructions,
meta-prompting guides the LM to break down complex tasks into smaller, more
manageable subtasks. These subtasks are then handled by distinct "expert"
instances of the same LM, each operating under specific, tailored instructions.
Central to this process is the LM itself, in its role as the conductor, which
ensures seamless communication and effective integration of the outputs from
these expert models. It additionally employs its inherent critical thinking and
robust verification processes to refine and authenticate the end result. This
collaborative prompting approach empowers a single LM to simultaneously act as
a comprehensive orchestrator and a panel of diverse experts, significantly
enhancing its performance across a wide array of tasks. The zero-shot,
task-agnostic nature of meta-prompting greatly simplifies user interaction by
obviating the need for detailed, task-specific instructions. Furthermore, our
research demonstrates the seamless integration of external tools, such as a
Python interpreter, into the meta-prompting framework, thereby broadening its
applicability and utility. Through rigorous experimentation with GPT-4, we
establish the superiority of meta-prompting over conventional scaffolding
methods: When averaged across all tasks, including the Game of 24,
Checkmate-in-One, and Python Programming Puzzles, meta-prompting, augmented
with a Python interpreter functionality, surpasses standard prompting by 17.1%,
expert (dynamic) prompting by 17.3%, and multipersona prompting by 15.2%.

通过引入元提示技术，将单个语言模型转变为多面手指挥者，通过高级指示将复杂任务分解为可管理的子任务，从而提高语言模型的综合性能。元提示方法的任务无关性大大简化了用户交互，而与 Python 解释器等外部工具的无缝集成扩展了其适用性和实用性。在 GPT-4 上进行的实验表明，相比传统的支架方法，使用元提示技术在所有任务上平均效果提高了 17.1%，动态支架方法提高了 17.3%，多身份支架方法提高了 15.2%。