Large Language Models (LLMs) have been applied to Math Word Problems (MWPs)
with transformative impacts, revolutionizing how these complex problems are
approached and solved in various domains including educational settings.
However, the evaluation of these models often prioritizes final accuracy,
overlooking the crucial aspect of reasoning capabilities. This work addresses
this gap by focusing on the ability of LLMs to detect and correct reasoning
mistakes. We introduce a novel dataset MWP-MISTAKE, incorporating MWPs with
both correct and incorrect reasoning steps generated through rule-based methods
and smaller language models. Our comprehensive benchmarking reveals significant
insights into the strengths and weaknesses of state-of-the-art models, such as
GPT-4o, GPT-4, GPT-3.5Turbo, and others. We highlight GPT-$o's superior
performance in mistake detection and rectification and the persistent
challenges faced by smaller models. Additionally, we identify issues related to
data contamination and memorization, impacting the reliability of LLMs in
real-world applications. Our findings emphasize the importance of rigorous
evaluation of reasoning processes and propose future directions to enhance the
generalization and robustness of LLMs in mathematical problem-solving.

通过引入一个包含正确和错误推理步骤的新数据集 MWP-MISTAKE，本研究重点探讨了大型语言模型在检测和纠正推理错误方面的能力，并通过全面的基准测试揭示了目前最先进模型（如 GPT-4o，GPT-4，GPT-3.5Turbo 等）的优点和缺点。此外，我们还发现涉及数据污染和记忆的问题，影响了大型语言模型在实际应用中的可靠性，因此强调了对推理过程进行严格评估的重要性，并提出了提高大型语言模型在数学问题解决中泛化性和鲁棒性的未来方向。