As LLMs have become increasingly popular, they have been used in almost every
field. But as the application for LLMs expands from generic fields to narrow,
focused science domains, there exists an ever-increasing gap in ways to
evaluate their efficacy in those fields. For the benchmarks that do exist, a
lot of them focus on questions that don't require proper understanding of the
subject in question. In this paper, we present NuclearQA, a human-made
benchmark of 100 questions to evaluate language models in the nuclear domain,
consisting of a varying collection of questions that have been specifically
designed by experts to test the abilities of language models. We detail our
approach and show how the mix of several types of questions makes our benchmark
uniquely capable of evaluating models in the nuclear domain. We also present
our own evaluation metric for assessing LLM's performances due to the
limitations of existing ones. Our experiments on state-of-the-art models
suggest that even the best LLMs perform less than satisfactorily on our
benchmark, demonstrating the scientific knowledge gap of existing LLMs.

这篇论文介绍了一种在核能领域评估语言模型性能的人工测试标准 ——NuclearQA，它由专家设计的 100 个问题组成，针对语言模型的能力进行了测试。论文还提出了一种新的评估指标，发现现有的最优语言模型在该测试标准上的表现不尽人意，揭示了现有语言模型的科学知识差距。