Large language models (LLMs) benefit greatly from prompt engineering, with
in-context learning standing as a pivital technique. While former approaches
have provided various ways to construct the demonstrations used for in-context
learning, they often ignore the inherent heterogeneity within datasets,
applying the same demonstrations to all reasoning questions. We observed that
the effectiveness of demonstrations varies depending on the specific question.
This motivates our exploration of using prompt engineering to select
appropriate demonstrations. To address the challenge of automatically creating
and choosing demonstrations tailored to each question, we propose a novel
prompt engineering workflow built around a novel object called the
"demonstration notebook." This notebook helps identify the most suitable
in-context learning example for a question by gathering and reusing information
from the LLM's past interactions. Our experiments show that this approach
outperforms all existing methods for automatic demonstration construction and
selection (as far as we know), achieving state-of-the-art results on serveral
reasoning benchmarks. The method's versatility is further demonstrated by its
success in text summarization and prompt compression tasks. Additionally, we
contribute a rigorous analysis method to reveal the "demonstrative regime" of a
demonstration, providing valuable insights into how demonstrations relate to
different question types within a dataset.

使用新颖的 “演示笔记本” 对象，通过从语言模型过去的交互中收集和重复使用信息来选择问题的适当示例，从而实现了自动演示构建和选择的方法，并在多个推理基准上取得了最先进的结果。同时，通过对演示的严格分析方法，揭示了不同问题类型之间演示的相关性，为问题的推理解决提供了有价值的见解。