In this work, we introduce a comprehensive error typology specifically
designed for evaluating two distinct tasks in machine-generated patent texts:
claims-to-abstract generation, and the generation of the next claim given
previous ones. We have also developed a benchmark, PatentEval, for
systematically assessing language models in this context. Our study includes a
comparative analysis, annotated by humans, of various models. These range from
those specifically adapted during training for tasks within the patent domain
to the latest general-purpose large language models (LLMs). Furthermore, we
explored and evaluated some metrics to approximate human judgments in patent
text evaluation, analyzing the extent to which these metrics align with expert
assessments. These approaches provide valuable insights into the capabilities
and limitations of current language models in the specialized field of patent
text generation.

我们介绍了一种专门设计用于评估机器生成专利文本中两个不同任务的全面错误类型学，即声明到摘要的生成以及给定前面声明的生成下一个声明。我们还开发了一个用于系统评估语言模型在此背景下的基准测试工具 PatentEval。此外，我们还对各种模型进行了人工标注的比较分析，这些模型从专门为专利领域内的任务而进行训练的模型到最新的通用大型语言模型（LLMs）。此外，我们还探索和评估了一些指标来近似专利文本评估中的人工判断，并分析这些指标在专家评估中的一致性程度。这些方法对当前语言模型在专利文本生成领域的能力和局限性提供了有价值的见解。