The primary way of building AI applications is shifting from training
specialist models to prompting generalist models. A common practice for
prompting generalist models, often referred to as in-context learning, is to
append a few examples (demonstrations) to the prompt to help the model better
understand the task. While effective, in-context learning can be inefficient
because it makes the input prompt much longer, consuming valuable space in the
context window and leading to larger computational costs. In this paper, we
propose DynaICL, a recipe for efficient prompting with black-box generalist
models that dynamically allocate in-context examples according to the input
complexity and the computational budget. To achieve this, we train a meta
controller that predicts the number of in-context examples suitable for the
generalist model to make a good prediction based on the performance-efficiency
trade-off for a specific input. We then dynamically allocate the number of
demonstrations for an input according to predictions from the meta controller
and the given computation budget. Experimental results show that dynamic
example allocation helps achieve a better performance-efficiency trade-off in
two practical settings where computational resources or the required
performance is constrained. Specifically, DynaICL saves up to 46% token budget
compared to the common practice that allocates the same number of in-context
examples to each input. We also find that a meta controller trained on a
certain backbone model and tasks can successfully generalize to unseen models
and tasks.

本文提出 DynaICL，基于黑盒通用模型的高效提示配方。通过动态分配示例来平衡性能和效率。实验结果表明 DynaICL 在计算资源或所需性能受限时可以实现更好的性能和效率权衡。