Scaling high-quality tutoring is a major challenge in education. Because of
the growing demand, many platforms employ novice tutors who, unlike
professional educators, struggle to effectively address student mistakes and
thus fail to seize prime learning opportunities for students. In this paper, we
explore the potential for large language models (LLMs) to assist math tutors in
remediating student mistakes. We present ReMath, a benchmark co-developed with
experienced math teachers that deconstructs their thought process for
remediation. The benchmark consists of three step-by-step tasks: (1) infer the
type of student error, (2) determine the strategy to address the error, and (3)
generate a response that incorporates that information. We evaluate the
performance of state-of-the-art instruct-tuned and dialog models on ReMath. Our
findings suggest that although models consistently improve upon original tutor
responses, we cannot rely on models alone to remediate mistakes. Providing
models with the error type (e.g., the student is guessing) and strategy (e.g.,
simplify the problem) leads to a 75% improvement in the response quality over
models without that information. Nonetheless, despite the improvement, the
quality of the best model's responses still falls short of experienced math
teachers. Our work sheds light on the potential and limitations of using
current LLMs to provide high-quality learning experiences for both tutors and
students at scale. Our work is open-sourced at this link:
https://github.com/rosewang2008/remath.

我们探索了大型语言模型在辅导教师补救学生错误中的潜力，并通过评估表明，尽管模型不断改进了原始辅导响应，但仍无法达到经验丰富的数学教师的水平。这项工作揭示了当前大型语言模型在提供规模化高质量学习体验方面的潜力和局限性。

学生数学错误逐步纠正

Step-by-Step Remediation of Students' Mathematical Mistakes

Large language models are becoming increasingly integrated into our lives.
Hence, it is important to understand the biases present in their outputs in
order to avoid perpetuating harmful stereotypes, which originate in our own
flawed ways of thinking. This challenge requires developing new benchmarks and
methods for quantifying affective and semantic bias, keeping in mind that LLMs
act as psycho-social mirrors that reflect the views and tendencies that are
prevalent in society. One such tendency that has harmful negative effects is
the global phenomenon of anxiety toward math and STEM subjects. Here, we
investigate perceptions of math and STEM fields provided by cutting-edge
language models, namely GPT-3, Chat-GPT, and GPT-4, by applying an approach
from network science and cognitive psychology. Specifically, we use behavioral
forma mentis networks (BFMNs) to understand how these LLMs frame math and STEM
disciplines in relation to other concepts. We use data obtained by probing the
three LLMs in a language generation task that has previously been applied to
humans. Our findings indicate that LLMs have an overall negative perception of
math and STEM fields, with math being perceived most negatively. We observe
significant differences across the three LLMs. We observe that newer versions
(i.e. GPT-4) produce richer, more complex perceptions as well as less negative
perceptions compared to older versions and N=159 high-school students. These
findings suggest that advances in the architecture of LLMs may lead to
increasingly less biased models that could even perhaps someday aid in reducing
harmful stereotypes in society rather than perpetuating them.

应用行为形态网络方法，研究了三个最新版本的大型语言模型 GPT-3、Chat-GPT 和 GPT-4 对数学和 STEM 领域的感知。结果表明，LLMs 对数学和 STEM 领域普遍持有负面看法，其中数学的看法尤为负面。最新版本的 LLMs 相对于旧版本和高中生感知更复杂、更少负面看法，这表明 LLMs 架构的进步有可能导致越来越不带偏见的模型，甚至有望消除有害刻板印象而不是延续它们。