New Natural Langauge Process~(NLP) benchmarks are urgently needed to align
with the rapid development of large language models (LLMs). We present Xiezhi,
the most comprehensive evaluation suite designed to assess holistic domain
knowledge. Xiezhi comprises multiple-choice questions across 516 diverse
disciplines ranging from 13 different subjects with 220,000 questions and
accompanied by Xiezhi-Specialty and Xiezhi-Interdiscipline, both with 15k
questions. We conduct evaluation of the 47 cutting-edge LLMs on Xiezhi. Results
indicate that LLMs exceed average performance of humans in science,
engineering, agronomy, medicine, and art, but fall short in economics,
jurisprudence, pedagogy, literature, history, and management. We anticipate
Xiezhi will help analyze important strengths and shortcomings of LLMs, and the
benchmark is released in this https URL .

我们提出了 Xiezhi，这是最全面的评估套件，包括了 516 个不同学科的 220,000 个多项选择题，旨在为检验大型语言模型的整体域知识提供帮助，并在新的自然语言处理基准中，对 47 个先进的大型语言模型进行了评估，它们在科学、工程、农学、医学和艺术领域超过了人类的平均表现，但在经济学、法学、教育学、文学、历史和管理学方面表现不佳。