Recent advancements in large language models (LLMs) have substantially
enhanced their mathematical reasoning abilities. However, these models still
struggle with complex problems that require multiple reasoning steps,
frequently leading to logical or numerical errors. While numerical mistakes can
largely be addressed by integrating a code interpreter, identifying logical
errors within intermediate steps is more challenging. Moreover, manually
annotating these steps for training is not only expensive but also demands
specialized expertise. In this study, we introduce an innovative approach that
eliminates the need for manual annotation by leveraging the Monte Carlo Tree
Search (MCTS) framework to generate both the process supervision and evaluation
signals automatically. Essentially, when a LLM is well pre-trained, only the
mathematical questions and their final answers are required to generate our
training data, without requiring the solutions. We proceed to train a
step-level value model designed to improve the LLM's inference process in
mathematical domains. Our experiments indicate that using automatically
generated solutions by LLMs enhanced with MCTS significantly improves the
model's proficiency in dealing with intricate mathematical reasoning tasks.

利用 Monte Carlo Tree Search (MCTS) 框架和大型语言模型 (LLMs) 自动生成解决步骤和评估信号，训练一个逐步价值模型以提高 LLM 在数学领域中的推理过程。实验证明，采用 LLMs 结合 MCTS 自动生成的解决方案显著提高了模型处理复杂数学推理任务的能力。