The recent LLMs like GPT-4 and PaLM-2 have made tremendous progress in
solving fundamental math problems like GSM8K by achieving over 90\% accuracy.
However, their capabilities to solve more challenging math problems which
require domain-specific knowledge (i.e. theorem) have yet to be investigated.
In this paper, we introduce TheoremQA, the first theorem-driven
question-answering dataset designed to evaluate AI models' capabilities to
apply theorems to solve challenging science problems. \dataset is curated by
domain experts containing 800 high-quality questions covering 350
theorems\footnote{e.g. Taylor's theorem, Lagrange's theorem, Huffman coding,
Quantum Theorem, Elasticity Theorem, etc} from Math, Physics, EE\&CS, and
Finance. We evaluate a wide spectrum of 16 large language and code models with
different prompting strategies like Chain-of-Thoughts and Program-of-Thoughts.
We found that GPT-4's capabilities to solve these problems are unparalleled,
achieving an accuracy of 51\% with Program-of-Thoughts Prompting. All the
existing open-sourced models are below 15\%, barely surpassing the random-guess
baseline. Given the diversity and broad coverage of \dataset, we believe it can
be used as a better benchmark to evaluate LLMs' capabilities to solve
challenging science problems. The data and code are released in
this https URL

本文介绍了一个以定理驱动的问答数据集 TheoremQA，用于评估人工智能模型应用定理解决具有挑战性的科学问题的能力。研究人员使用 16 个大型语言和代码模型评估 TheoremQA，并发现 GPT-4 在 Program-of-Thoughts Prompting 的帮助下解决这些问题的能力是无与伦比的，达到了 51％，而现有的所有开放源代码模型都低于 15％，仅仅超过了随机猜测的基线。