In this work, we propose a novel method named \textbf{Auto}mated Process
Labeling via \textbf{C}onfidence \textbf{V}ariation (\textbf{\textsc{AutoCV}})
to enhance the reasoning capabilities of large language models (LLMs) by
automatically annotating the reasoning steps. Our approach begins by training a
verification model on the correctness of final answers, enabling it to generate
automatic process annotations. This verification model assigns a confidence
score to each reasoning step, indicating the probability of arriving at the
correct final answer from that point onward. We detect relative changes in the
verification's confidence scores across reasoning steps to automatically
annotate the reasoning process. This alleviates the need for numerous manual
annotations or the high computational costs associated with model-induced
annotation approaches. We experimentally validate that the confidence
variations learned by the verification model trained on the final answer
correctness can effectively identify errors in the reasoning steps.
Subsequently, we demonstrate that the process annotations generated by
\textsc{AutoCV} can improve the accuracy of the verification model in selecting
the correct answer from multiple outputs generated by LLMs. Notably, we achieve
substantial improvements across five datasets in mathematics and commonsense
reasoning. The source code of \textsc{AutoCV} is available at
https://github.com/rookie-joe/AUTOCV.

我们提出了一种名为 AutoCV 的新方法，通过自动注释推理步骤来增强大型语言模型的推理能力。我们的方法通过训练一个验证模型来为最终答案的正确性评估生成自动过程注释，并且通过检测验证模型在推理步骤中的置信度变化来自动注释推理过程。我们的实验证明，验证模型学习到的置信度变化可以有效地识别推理步骤中的错误，并且通过 AutoCV 生成的过程注释可以提高验证模型在多个输出中选择正确答案的准确性。