The derivation of mathematical results in specialised fields using Large
Language Models (LLMs) is an emerging research direction that can help identify
models' limitations, and potentially support mathematical discovery. In this
paper, we leverage a symbolic engine to generate derivations of equations at
scale, and investigate the capabilities of LLMs when deriving goal equations
from premises. Specifically, we employ in-context learning for GPT and
fine-tune a range of T5 models to compare the robustness and generalisation of
pre-training strategies to specialised models. Empirical results show that
fine-tuned FLAN-T5-large (MathT5) outperforms GPT models on all static and
out-of-distribution test sets in terms of absolute performance. However, an
in-depth analysis reveals that the fine-tuned models are more sensitive to
perturbations involving unseen symbols and (to a lesser extent) changes to
equation structure. In addition, we analyse 1.7K equations and over 200
derivations to highlight common reasoning errors such as the inclusion of
incorrect, irrelevant, and redundant equations, along with the tendency to skip
derivation steps. Finally, we explore the suitability of existing metrics for
evaluating mathematical derivations finding evidence that, while they capture
general properties such as sensitivity to perturbations, they fail to highlight
fine-grained reasoning errors and essential differences between models.
Overall, this work demonstrates that training models on synthetic data can
improve their mathematical capabilities beyond larger architectures.

利用大型语言模型 (LLMs) 进行特定领域的数学推导是一项新兴的研究方向，可以帮助发现模型的局限性，潜在地支持数学发现。本文利用符号引擎在大规模上生成方程的推导，并研究 LLMs 在从前提中推导目标方程时的能力。实证结果表明，fine-tuned FLAN-T5-large (MathT5) 在绝对性能方面超过了 GPT 模型在所有静态和分布外的测试集上。然而，深入分析发现，fine-tuned 模型对于涉及未知符号的扰动 (以及较小程度的方程结构更改) 更为敏感。此外，我们分析了 1.7K 个方程和 200 多个推导，突出了常见的推理错误，如包含不正确、无关或多余的方程，以及跳过推导步骤的倾向。最后，我们探讨了评估数学推导的现有指标的适用性，发现尽管它们能捕捉到对扰动的敏感性等一般属性，但无法凸显细粒度的推理错误和模型之间的重要差异。总体来说，本研究表明在合成数据上训练模型可以提高其数学能力，超越更大的架构。