As Large Language Models (LLMs) are becoming prevalent in various fields,
there is an urgent need for improved NLP benchmarks that encompass all the
necessary knowledge of individual discipline. Many contemporary benchmarks for
foundational models emphasize a broad range of subjects but often fall short in
presenting all the critical subjects and encompassing necessary professional
knowledge of them. This shortfall has led to skewed results, given that LLMs
exhibit varying performance across different subjects and knowledge areas. To
address this issue, we present psybench, the first comprehensive Chinese
evaluation suite that covers all the necessary knowledge required for graduate
entrance exams. psybench offers a deep evaluation of a model's strengths and
weaknesses in psychology through multiple-choice questions. Our findings show
significant differences in performance across different sections of a subject,
highlighting the risk of skewed results when the knowledge in test sets is not
balanced. Notably, only the ChatGPT model reaches an average accuracy above
$70\%$, indicating that there is still plenty of room for improvement. We
expect that psybench will help to conduct thorough evaluations of base models'
strengths and weaknesses and assist in practical application in the field of
psychology.

LLMs 在各领域越来越普及，因此迫切需要改进的 NLP 基准来涵盖各个学科的必要知识。为了解决这个问题，作者提出了 psybench，这是第一个详尽覆盖研究领域所需知识的全面中文评估套件，通过多项选择题深入评估模型在心理学中的优势和劣势，并发现不同领域的性能存在显著差异，展示了测试集中知识不均衡可能导致结果偏斜的风险，仅 ChatGPT 模型的平均准确率超过 70％，表明仍有很大的改进空间。总之，psybench 将有助于对基本模型的优势和劣势进行全面评估，并在心理学领域的实际应用中提供支持。