Mathematical problem solving is an important skill for Large Language Models
(LLMs), both as an important capability and a proxy for a range of reasoning
abilities. Existing benchmarks probe a diverse set of skills, but they yield
aggregate accuracy metrics, obscuring specific abilities or weaknesses.
Furthermore, they are difficult to extend with new problems, risking data
contamination over time. To address these challenges, we propose MathCAMPS: a
method to synthesize high-quality mathematical problems at scale, grounded on
44 fine-grained "standards" from the Mathematics Common Core (CC) Standard for
K-8 grades. We encode each standard in a formal grammar, allowing us to sample
diverse symbolic problems and their answers. We then use LLMs to realize the
symbolic problems into word problems. We propose a cycle-consistency method for
validating problem faithfulness. Finally, we derive follow-up questions from
symbolic structures and convert them into follow-up word problems - a novel
task of mathematical dialogue that probes for robustness in understanding.
Experiments on 23 LLMs show surprising failures even in the strongest models
(in particular when asked simple follow-up questions). Moreover, we evaluate
training checkpoints of Pythia 12B on MathCAMPS, allowing us to analyze when
particular mathematical skills develop during its training. Our framework
enables the community to reproduce and extend our pipeline for a fraction of
the typical cost of building new high-quality datasets.

我们提出了 MathCAMPS 方法，用于合成高质量的数学问题，并通过 LLMs 将其转化为单词问题，以进一步探索数学问题解决和对话中的各种技能和能力。

MathCAMPS: 从人类课程中细化合成数学问题

MathCAMPS: Fine-grained Synthesis of Mathematical Problems From Human  Curricula

Many intellectual endeavors require mathematical problem solving, but this
skill remains beyond the capabilities of computers. To measure this ability in
machine learning models, we introduce MATH, a new dataset of 12,500 challenging
competition mathematics problems. Each problem in MATH has a full step-by-step
solution which can be used to teach models to generate answer derivations and
explanations. To facilitate future research and increase accuracy on MATH, we
also contribute a large auxiliary pretraining dataset which helps teach models
the fundamentals of mathematics. Even though we are able to increase accuracy
on MATH, our results show that accuracy remains relatively low, even with
enormous Transformer models. Moreover, we find that simply increasing budgets
and model parameter counts will be impractical for achieving strong
mathematical reasoning if scaling trends continue. While scaling Transformers
is automatically solving most other text-based tasks, scaling is not currently
solving MATH. To have more traction on mathematical problem solving we will
likely need new algorithmic advancements from the broader research community.

介绍了一种新的 MATH 数据集，其中包含 12500 个具有挑战性的竞赛数学问题，每个问题都有全面的解决方案，可以用于教模型生成答案推导和解释。尽管研究者不断增加 Transformer 模型的预算和参数总数，但在 MATH 的准确性仍然相对较低，因此未来的研究可能需要更广泛的算法进步。