Modern large language models (LLMs) have demonstrated impressive capabilities
at sophisticated tasks, often through step-by-step reasoning similar to humans.
This is made possible by their strong few and zero-shot abilities -- they can
effectively learn from a handful of handcrafted, completed responses
("in-context examples"), or are prompted to reason spontaneously through
specially designed triggers. Nonetheless, some limitations have been observed.
First, performance in the few-shot setting is sensitive to the choice of
examples, whose design requires significant human effort. Moreover, given the
diverse downstream tasks of LLMs, it may be difficult or laborious to handcraft
per-task labels. Second, while the zero-shot setting does not require
handcrafting, its performance is limited due to the lack of guidance to the
LLMs. To address these limitations, we propose Consistency-based Self-adaptive
Prompting (COSP), a novel prompt design method for LLMs. Requiring neither
handcrafted responses nor ground-truth labels, COSP selects and builds the set
of examples from the LLM zero-shot outputs via carefully designed criteria that
combine consistency, diversity and repetition. In the zero-shot setting for
three different LLMs, we show that using only LLM predictions, COSP improves
performance up to 15% compared to zero-shot baselines and matches or exceeds
few-shot baselines for a range of reasoning tasks.

本研究提出了一种基于一致性的自适应提示设计方法，可以从大型语言模型的零样本输出中选择和构建示例，进而显著提高了零样本情况下三种不同大型语言模型的推理任务的性能。