Large language models (LLMs) demonstrate substantial capabilities in solving
math problems. However, they tend to produce hallucinations when given
questions containing unreasonable errors. In this paper, we study the behavior
of LLMs when faced with unreasonable math problems and further explore their
potential to address these problems. First, we construct the Unreasonable Math
Problem (UMP) benchmark to examine the error detection ability of LLMs.
Experiments show that LLMs are able to detect unreasonable errors, but still
fail in generating non-hallucinatory content. In order to improve their ability
of error detection and correction, we further design a strategic prompt
template called Critical Calculation and Conclusion(CCC). With CCC, LLMs can
better self-evaluate and detect unreasonable errors in math questions, making
them more reliable and safe in practical application scenarios.

大型语言模型在解决数学问题方面具有重大能力，但在面对包含不合理错误的问题时倾向于产生幻觉。本文研究了 LLM 在面对不合理数学问题时的行为，并进一步探讨了它们解决这些问题的潜力。实验表明，LLM 能够检测到不合理错误，但在生成非幻觉性内容方面仍然失败。为了提高它们的错误检测和修正能力，我们设计了一种战略提示模板，称为 Critical Calculation and Conclusion（CCC）。借助 CCC，LLM 可以更好地自我评估和检测数学问题中的不合理错误，使其在实际应用场景中更可靠和安全。