Classical Chinese Understanding (CCU) holds significant value in preserving
and exploration of the outstanding traditional Chinese culture. Recently,
researchers have attempted to leverage the potential of Large Language Models
(LLMs) for CCU by capitalizing on their remarkable comprehension and semantic
capabilities. However, no comprehensive benchmark is available to assess the
CCU capabilities of LLMs. To fill this gap, this paper introduces C$^{3}$bench,
a Comprehensive Classical Chinese understanding benchmark, which comprises
50,000 text pairs for five primary CCU tasks, including classification,
retrieval, named entity recognition, punctuation, and translation. Furthermore,
the data in C$^{3}$bench originates from ten different domains, covering most
of the categories in classical Chinese. Leveraging the proposed C$^{3}$bench,
we extensively evaluate the quantitative performance of 15 representative LLMs
on all five CCU tasks. Our results not only establish a public leaderboard of
LLMs' CCU capabilities but also gain some findings. Specifically, existing LLMs
are struggle with CCU tasks and still inferior to supervised models.
Additionally, the results indicate that CCU is a task that requires special
attention. We believe this study could provide a standard benchmark,
comprehensive baselines, and valuable insights for the future advancement of
LLM-based CCU research. The evaluation pipeline and dataset are available at
https://github.com/SCUT-DLVCLab/C3bench.

通过提出 C^3bench，我们在五个主要的古文理解任务中广泛评估了 15 个代表性大语言模型的定量性能。结果表明，现有的大语言模型在古文理解任务上仍然表现不佳，相比受监督模型仍然较差。我们相信这项研究可以为未来基于大语言模型的古文理解研究提供标准基准、全面的基线和有价值的见解。