Recent advances in large language models (LLMs) have demonstrated notable
progress on many mathematical benchmarks. However, most of these benchmarks
only feature problems grounded in junior and senior high school subjects,
contain only multiple-choice questions, and are confined to a limited scope of
elementary arithmetic operations. To address these issues, this paper
introduces an expansive benchmark suite SciBench that aims to systematically
examine the reasoning capabilities required for complex scientific problem
solving. SciBench contains two carefully curated datasets: an open set
featuring a range of collegiate-level scientific problems drawn from
mathematics, chemistry, and physics textbooks, and a closed set comprising
problems from undergraduate-level exams in computer science and mathematics.
Based on the two datasets, we conduct an in-depth benchmark study of two
representative LLMs with various prompting strategies. The results reveal that
current LLMs fall short of delivering satisfactory performance, with an overall
score of merely 35.80%. Furthermore, through a detailed user study, we
categorize the errors made by LLMs into ten problem-solving abilities. Our
analysis indicates that no single prompting strategy significantly outperforms
others and some strategies that demonstrate improvements in certain
problem-solving skills result in declines in other skills. We envision that
SciBench will catalyze further developments in the reasoning abilities of LLMs,
thereby ultimately contributing to scientific research and discovery.

最近大型语言模型 (LLM) 在许多数学基准上取得显著进展，但大多数基准只涉及初高中科目的问题和多项选择题，且仅限于有限范围的基本算术操作。为解决这些问题，本文引入了一个广泛的基准套件 SciBench，旨在系统地检验复杂科学问题解决所需的推理能力。SciBench 包含两个精心策划的数据集：一个开放集，包含从数学、化学和物理教科书中提取的多领域的大学级科学问题；一个封闭集，包含了计算机科学和数学本科考试中的问题。基于这两个数据集，我们对两个代表性 LLM 进行了深入的基准研究，采用多种提示策略。结果表明，当前 LLM 的表现不尽如人意，综合得分仅为 35.80%。此外，通过详细的用户研究，我们将 LLM 的错误归类为十种问题解决能力。我们的分析表明，没有一种单一的提示策略明显优于其他策略，而且一些策略在某些问题解决技能上的提高会导致其他技能下降。我们预计 SciBench 将催生 LLM 的推理能力进一步发展，从而最终促进科学研究和发现。