In the context of multi-step reasoning, language models (LMs) probabilities
are often miscalibrated -- solutions with high probabilities are not always
correct. Therefore, greedy decoding, which is the standard decoding method for
reasoning tasks, often yields incorrect solutions. In addition, methods such as
self-consistency and verifiers rely on sampling from the LM distribution and do
not tackle the underlying issue. To address this, we introduce Guiding
Multi-step ReAsoning with a CorrectnEss Discriminator (GRACE), a stepwise
decoding approach that nudges the model towards producing correct reasoning
steps. GRACE employs a discriminator model, which is trained to differentiate
correct steps from invalid ones, to adjust decoding preferences based on the
correctness of each reasoning step. Importantly, GRACE does not require
fine-tuning or re-training the LMs. When compared with conventional decoding
strategies over four popular math reasoning benchmarks, GRACE exhibits
significant improvements in both final answer accuracy and step correctness,
outperforming both greedy decoding and self-consistency.\footnote{Our code can
be found at https://github.com/mukhal/grace.}

该研究提出了一种基于 stepwise decoding 方法的 Guiding Multi-step ReAsoning with a CorrectnEss Discriminator (GRACE) ，它使用判别模型来调整 LM 的解码策略，从而提高多步推理的准确性。与传统解码策略相比，GRACE 在四个流行的数学推理基准测试中均表现出显著的改进。