We introduce a novel framework, LM-Guided CoT, that leverages a lightweight
(i.e., <1B) language model (LM) for guiding a black-box large (i.e., >10B) LM
in reasoning tasks. Specifically, the lightweight LM first generates a
rationale for each input instance. The Frozen large LM is then prompted to
predict a task output based on the rationale generated by the lightweight LM.
Our approach is resource-efficient in the sense that it only requires training
the lightweight LM. We optimize the model through 1) knowledge distillation and
2) reinforcement learning from rationale-oriented and task-oriented reward
signals. We assess our method with multi-hop extractive question answering (QA)
benchmarks, HotpotQA, and 2WikiMultiHopQA. Experimental results show that our
approach outperforms all baselines regarding answer prediction accuracy. We
also find that reinforcement learning helps the model to produce higher-quality
rationales with improved QA performance.

我们引入了一种新颖的框架，LM-Guided CoT，它利用轻量级（即 <1B）语言模型（LM）指导黑盒大型（即> 10B）LM 在推理任务中的工作。具体而言，轻量级 LM 首先为每个输入实例生成理论依据，接着使用冻结的大型 LM 根据轻量级 LM 生成的理论依据预测任务输出。我们的方法在资源利用上高效，只需要训练轻量级 LM。我们通过知识蒸馏和强化学习来优化模型，其中蒸馏和任务导向的奖励信号用于强化学习。我们通过多跳摘要问答基准 HotpotQA 和 2WikiMultiHopQA 来评估我们的方法。实验结果表明，我们的方法在回答预测准确性方面优于所有基准方法。我们还发现，强化学习有助于模型生成更高质量的理论依据，并提高问答的性能。