With the emergence of numerous legal LLMs, there is currently a lack of a
comprehensive benchmark for evaluating their legal abilities. In this paper, we
propose the first Chinese Legal LLMs benchmark based on legal capabilities.
Through the collaborative efforts of legal and artificial intelligence experts,
we divide the legal capabilities of LLMs into three levels: basic legal NLP
capability, basic legal application capability, and complex legal application
capability. We have completed the first phase of evaluation, which mainly
focuses on the capability of basic legal NLP. The evaluation results show that
although some legal LLMs have better performance than their backbones, there is
still a gap compared to ChatGPT. Our benchmark can be found at URL.

我们提出了第一个基于法学能力的中国法学硕士 (LLMs) 综合评估基准。通过法律和人工智能专家的协作努力，我们将法学硕士的法律能力分为三个层次：基本法律自然语言处理能力、基本法律应用能力和复杂法律应用能力。我们已完成了第一阶段的评估，主要关注基本法律自然语言处理能力。评估结果显示，尽管一些法学硕士在性能上优于它们的基础模型，但与 ChatGPT 相比仍存在差距。我们的基准测试可以在 URL 上找到。