We investigate the mathematical capabilities of ChatGPT by testing it on
publicly available datasets, as well as hand-crafted ones, and measuring its
performance against other models trained on a mathematical corpus, such as
Minerva. We also test whether ChatGPT can be a useful assistant to professional
mathematicians by emulating various use cases that come up in the daily
professional activities of mathematicians (question answering, theorem
searching). In contrast to formal mathematics, where large databases of formal
proofs are available (e.g., the Lean Mathematical Library), current datasets of
natural-language mathematics, used to benchmark language models, only cover
elementary mathematics. We address this issue by introducing a new dataset:
GHOSTS. It is the first natural-language dataset made and curated by working
researchers in mathematics that (1) aims to cover graduate-level mathematics
and (2) provides a holistic overview of the mathematical capabilities of
language models. We benchmark ChatGPT on GHOSTS and evaluate performance
against fine-grained criteria. We make this new dataset publicly available to
assist a community-driven comparison of ChatGPT with (future) large language
models in terms of advanced mathematical comprehension. We conclude that
contrary to many positive reports in the media (a potential case of selection
bias), ChatGPT's mathematical abilities are significantly below those of an
average mathematics graduate student. Our results show that ChatGPT often
understands the question but fails to provide correct solutions. Hence, if your
goal is to use it to pass a university exam, you would be better off copying
from your average peer!

本研究使用 GHOSTS 数据集评估了 ChatGPT 的数学能力和其他训练过数学语料库的模型相比，发现其数学能力显著低于普通数学研究生，并强调 GHOSTS 数据集的重要性以及未来大型语言模型在高级数学理解方面的比较研究。