We introduce a new framework, Directional Stimulus Prompting, that uses a
tuneable language model (LM) to provide guidance for the black-box frozen large
language model (LLM) on downstream tasks. Unlike prior work that manually or
automatically finds the optimal prompt for each task, we train a policy LM to
generate discrete tokens as ``directional stimulus'' of each input, which is a
hint/cue such as keywords of an article for summarization. The directional
stimulus is then combined with the original input and fed into the LLM to guide
its generation toward the desired target. The policy LM can be trained through
1) supervised learning from annotated data and 2) reinforcement learning from
offline and online rewards to explore directional stimulus that better aligns
LLMs with human preferences. This framework is flexibly applicable to various
LMs and tasks. To verify its effectiveness, we apply our framework to
summarization and dialogue response generation tasks. Experimental results
demonstrate that it can significantly improve LLMs' performance with a small
collection of training data: a T5 (780M) trained with 2,000 samples from the
CNN/Daily Mail dataset improves Codex (175B)'s performance by 7.2% in ROUGE-Avg
scores; 500 dialogues boost the combined score by 52.5%, achieving comparable
or even better performance than fully trained models on the MultiWOZ dataset.

通过引入 “方向刺激提示” 框架，使用可调的语言模型（LM）为黑盒冷冻的大语言模型（LLM）在下游任务中提供指导，以重点提示 / 示范以及方向性刺激等手段协助 LLM 完成任务，通过验证取得了鼓舞人心的提升。