The rapid development of Large Language Models (LLMs) in vertical domains,
including intellectual property (IP), lacks a specific evaluation benchmark for
assessing their understanding, application, and reasoning abilities. To fill
this gap, we introduce IPEval, the first evaluation benchmark tailored for IP
agency and consulting tasks. IPEval comprises 2657 multiple-choice questions
across four major dimensions: creation, application, protection, and management
of IP. These questions span patent rights (inventions, utility models,
designs), trademarks, copyrights, trade secrets, and other related laws.
Evaluation methods include zero-shot, 5-few-shot, and Chain of Thought (CoT)
for seven LLM types, predominantly in English or Chinese. Results show superior
English performance by models like GPT series and Qwen series, while
Chinese-centric LLMs excel in Chinese tests, albeit specialized IP LLMs lag
behind general-purpose ones. Regional and temporal aspects of IP underscore the
need for LLMs to grasp legal nuances and evolving laws. IPEval aims to
accurately gauge LLM capabilities in IP and spur development of specialized
models. Website: https://ipeval.github.io/

鉴于在知识产权领域中大型语言模型（LLMs）的快速发展，缺乏一种特定的评估基准来评估其理解、应用和推理能力，我们引入了 IPEval，这是第一个为知识产权机构和咨询任务量身定制的评估基准。IPEval 包括 2657 个多项选择题，涵盖了知识产权的创造、应用、保护和管理等四个主要方面。这些问题涉及专利权（发明、实用新型、设计）、商标、版权、商业秘密和其他相关法律。评估方法包括零 - shot、5 - 少量 - shot 和 CoT 等多种类型的 LLM，主要以英文或中文为主。结果显示，GPT 系列和 Qwen 系列等模型在英文测试中表现优异，而以中文为主的 LLM 在中文测试中表现出色，尽管专门的鉴于知识产权的 LLMs 落后于通用性模型。知识产权的地区和时间方面强调了 LLMs 掌握法律细微差别和不断发展的法律的需求。IPEval 旨在准确评估 LLMs 在知识产权方面的能力，并推动专门模型的发展。