Chain of Thought (CoT) is significant in improving the reasoning abilities of
large language models (LLMs). However, the correlation between the
effectiveness of CoT and the length of reasoning steps in prompts remains
largely unknown. To shed light on this, we have conducted several empirical
experiments to explore the relations. Specifically, we design experiments that
expand and compress the rationale reasoning steps within CoT demonstrations,
while keeping all other factors constant. We have the following key findings.
First, the results indicate that lengthening the reasoning steps in prompts,
even without adding new information into the prompt, considerably enhances
LLMs' reasoning abilities across multiple datasets. Alternatively, shortening
the reasoning steps, even while preserving the key information, significantly
diminishes the reasoning abilities of models. This finding highlights the
importance of the number of steps in CoT prompts and provides practical
guidance to make better use of LLMs' potential in complex problem-solving
scenarios. Second, we also investigated the relationship between the
performance of CoT and the rationales used in demonstrations. Surprisingly, the
result shows that even incorrect rationales can yield favorable outcomes if
they maintain the requisite length of inference. Third, we observed that the
advantages of increasing reasoning steps are task-dependent: simpler tasks
require fewer steps, whereas complex tasks gain significantly from longer
inference sequences.

通过一系列实验，研究发现链式思维（CoT）在大型语言模型（LLMs）的推理能力提升中起到重要作用，推理步骤的长度与 CoT 的有效性之间的相关性还不完全清楚，本研究对此进行了探索，并得出了以下主要发现：1. 增加问题中推理步骤的长度能显著提高 LLMs 在多个数据集上的推理能力；2. 即使使用的推理方法有误，只要保持了必要的推理步骤长度，仍能取得良好的结果；3. 推理步骤的增加对任务的优势取决于任务的复杂性。这些发现对于在复杂问题解决场景中更好地利用 LLMs 的潜力提供了实际指导。