In light of the rapidly evolving capabilities of large language models
(LLMs), it becomes imperative to develop rigorous domain-specific evaluation
benchmarks to accurately assess their capabilities. In response to this need,
this paper introduces ArcMMLU, a specialized benchmark tailored for the Library
& Information Science (LIS) domain in Chinese. This benchmark aims to measure
the knowledge and reasoning capability of LLMs within four key sub-domains:
Archival Science, Data Science, Library Science, and Information Science.
Following the format of MMLU/CMMLU, we collected over 6,000 high-quality
questions for the compilation of ArcMMLU. This extensive compilation can
reflect the diverse nature of the LIS domain and offer a robust foundation for
LLM evaluation. Our comprehensive evaluation reveals that while most mainstream
LLMs achieve an average accuracy rate above 50% on ArcMMLU, there remains a
notable performance gap, suggesting substantial headroom for refinement in LLM
capabilities within the LIS domain. Further analysis explores the effectiveness
of few-shot examples on model performance and highlights challenging questions
where models consistently underperform, providing valuable insights for
targeted improvements. ArcMMLU fills a critical gap in LLM evaluations within
the Chinese LIS domain and paves the way for future development of LLMs
tailored to this specialized area.

该研究介绍了 ArcMMLU，这是一种专门为中文图书馆与信息科学领域定制的基准测试，它旨在衡量大型语言模型在档案学、数据科学、图书馆学和信息科学四个子领域内的知识和推理能力。通过对超过 6,000 个高质量问题的收集，ArcMMLU 构成了一个广泛的编译，能够反映 LIS 领域的多样性并为 LLM 评估提供可靠的基础。该研究发现，虽然大多数主流 LLM 在 ArcMMLU 上的平均准确率超过 50％，但仍然存在显著的性能差距，表明 LIS 领域的 LLM 功能仍有提升的空间。进一步分析探讨了少样本示例对模型性能的影响，并突出了模型在一些具有挑战性问题上的持续低效表现，为有针对性的改进提供了有价值的洞见。ArcMMLU 填补了中文 LIS 领域 LLM 评估中的一个重要空白，为未来定制该专门领域的 LLM 的发展铺平了道路。