Drori et al. (2022) report that "A neural network solves, explains, and
generates university math problems by program synthesis and few-shot learning
at human level ... [It] automatically answers 81\% of university-level
mathematics problems." The system they describe is indeed impressive; however,
the above description is very much overstated. The work of solving the problems
is done, not by a neural network, but by the symbolic algebra package Sympy.
Problems of various formats are excluded from consideration. The so-called
"explanations" are just rewordings of lines of code. Answers are marked as
correct that are not in the form specified in the problem. Most seriously, it
seems that in many cases the system uses the correct answer given in the test
corpus to guide its path to solving the problem.

Drori 等人（2022 年）使用符号代数包 Sympy 而非神经网络解决了大学数学问题，并通过计划合成和少量学习回答了 81％的问题，此外，它的 “解释” 仅是代码行的改写且在某些情况下似乎使用测试语料库中给定的答案来指导其解决问题的步骤。

一款解决大学数学问题的 AI 程序的限制

Limits of an AI program for solving college math problems

We present an open source math-aware Question Answering System based on Ask
Platypus. Our system returns as a single mathematical formula for a natural
language question in English or Hindi. This formulae originate from the
knowledge-base Wikidata. We translate these formulae to computable data by
integrating the calculation engine sympy into our system. This way, users can
enter numeric values for the variables occurring in the formula. Moreover, the
system loads numeric values for constants occurring in the formula from
Wikidata. In a user study, our system outperformed a commercial computational
mathematical knowledge engine by 13%. However, the performance of our system
heavily depends on the size and quality of the formula data available in
Wikidata. Since only a few items in Wikidata contained formulae when we started
the project, we facilitated the import process by suggesting formula edits to
Wikidata editors. With the simple heuristic that the first formula is
significant for the article, 80% of the suggestions were correct.

我们提出了一种基于 Ask Platypus 的开源数学感知问答系统，它能够针对英语或印地语的自然语言问题返回单一的数学公式。这些公式来源于知识库 Wikidata，并通过 sympy 计算引擎将其转换为可计算的数据，用户可以为公式中出现的变量输入数值。在用户研究中，我们的系统性能优于商用计算数学知识引擎 13％。