Chain-of-thought (CoT) prompting, which offers step-by-step problem-solving
rationales, has impressively unlocked the reasoning potential of large language
models (LLMs). Yet, the standard CoT is less effective in problems demanding
multiple reasoning steps. This limitation arises from the complex reasoning
process in multi-step problems: later stages often depend on the results of
several steps earlier, not just the results of the immediately preceding step.
Such complexities suggest the reasoning process is naturally represented as a
graph. The almost linear and straightforward structure of CoT prompting,
however, struggles to capture this complex reasoning graph. To address this
challenge, we propose Residual Connection Prompting (RESPROMPT), a new
prompting strategy that advances multi-step reasoning in LLMs. Our key idea is
to reconstruct the reasoning graph within prompts. We achieve this by
integrating necessary connections-links present in the reasoning graph but
missing in the linear CoT flow-into the prompts. Termed "residual connections",
these links are pivotal in morphing the linear CoT structure into a graph
representation, effectively capturing the complex reasoning graphs inherent in
multi-step problems. We evaluate RESPROMPT on six benchmarks across three
diverse domains: math, sequential, and commonsense reasoning. For the
open-sourced LLaMA family of models, RESPROMPT yields a significant average
reasoning accuracy improvement of 12.5% on LLaMA-65B and 6.8% on LLaMA2-70B.
Breakdown analysis further highlights RESPROMPT particularly excels in complex
multi-step reasoning: for questions demanding at least five reasoning steps,
RESPROMPT outperforms the best CoT based benchmarks by a remarkable average
improvement of 21.1% on LLaMA-65B and 14.3% on LLaMA2-70B. Through extensive
ablation studies and analyses, we pinpoint how to most effectively build
residual connections.

通过在问题提示中集成残差连接，Residual Connection Prompting (RESPROMPT) 有效地捕捉了多步骤推理中固有的复杂推理图，为开源的 LLaMA 模型系列在数学、顺序和常识推理的六个基准测试上显著提高了推理准确性，并在至少需要五个推理步骤的问题中相比最好的基于 Chain-of-thought (CoT) 的基准测试提高了 21.1% 和 14.3%。