Recently, there has been growing interest in using Large Language Models (LLMs) for scientific research. Numerous benchmarks have been proposed to evaluate the ability of LLMs for scientific research. However, current benchmarks are mostly based on pre-collected objective questions. This design suffers from data leakage problem and lacks the evaluation of subjective Q/A ability. In this paper, we propose SciEval, a comprehensive and multi-disciplinary evaluation benchmark to address these issues. Based on Bloom's taxonomy, SciEval covers four dimensions to systematically evaluate scientific research ability. In particular, we design a "dynamic" subset based on scientific principles to prevent evaluation from potential data leakage. Both objective and subjective questions are included in SciEval. These characteristics make SciEval a more effective benchmark for scientific research ability evaluation of LLMs. Comprehensive experiments on most advanced LLMs show that, although GPT-4 achieves SOTA performance compared to other LLMs, there is still substantial room for improvement, especially for dynamic questions. The data and codes are now publicly available.

这篇论文提出了SciEval基准评估体系，以解决现有预先收集客观问题的数据泄露问题和缺乏主观问答能力评估的问题。SciEval基于Bloom的认知分类学，覆盖了四个维度，系统评估科学研究能力。研究者进行了全面的实验证明，尽管GPT-4在与其他LLMs相比取得了最先进的表现，但仍有很大的改进空间，特别是在动态问题方面。数据和代码现已公开。

SciEval: 一个用于科学研究的多级大型语言模型评估基准