We propose novel evaluations for mathematical reasoning capabilities of Large
Language Models (LLMs) based on mathematical misconceptions. Our primary
approach is to simulate LLMs as a novice learner and an expert tutor, aiming to
identify the incorrect answer to math question resulted from a specific
misconception and to recognize the misconception(s) behind an incorrect answer,
respectively. Contrary to traditional LLMs-based mathematical evaluations that
focus on answering math questions correctly, our approach takes inspirations
from principles in educational learning sciences. We explicitly ask LLMs to
mimic a novice learner by answering questions in a specific incorrect manner
based on incomplete knowledge; and to mimic an expert tutor by identifying
misconception(s) corresponding to an incorrect answer to a question. Using
simple grade-school math problems, our experiments reveal that, while LLMs can
easily answer these questions correctly, they struggle to identify 1) the
incorrect answer corresponding to specific incomplete knowledge
(misconceptions); 2) the misconceptions that explain particular incorrect
answers. Our study indicates new opportunities for enhancing LLMs' math
reasoning capabilities, especially on developing robust student simulation and
expert tutoring models in the educational applications such as intelligent
tutoring systems.

我们提出了一种基于数学误解的大型语言模型（LLM）数学推理能力的新评估方法。我们通过模拟 LLMs 作为初学者和专家导师，旨在识别由于特定误解导致的错误答案，并识别一个错误答案背后的误解。与传统基于 LLMs 的数学评估侧重于正确回答数学问题不同，我们的方法受到教育学习科学原则的启发。我们要求 LLMs 明确地模仿初学者通过基于不完整知识的特定错误方式回答问题，并模仿专家导师识别与问题的错误答案相对应的误解。通过简单的小学数学问题实验，我们发现，尽管 LLMs 可以轻松正确回答这些问题，但它们难以识别：1）与特定不完整知识（误解）相对应的错误答案；2）解释特定错误答案的误解。我们的研究指出了增强 LLMs 数学推理能力的新机会，尤其是在教育应用中开发健壮的学生模拟和专家辅导模型方面。