Traditional evaluation metrics like ROUGE compare lexical overlap between the
reference and generated summaries without taking argumentative structure into
account, which is important for legal summaries. In this paper, we propose a
novel legal summarization evaluation framework that utilizes GPT-4 to generate
a set of question-answer pairs that cover main points and information in the
reference summary. GPT-4 is then used to generate answers based on the
generated summary for the questions from the reference summary. Finally, GPT-4
grades the answers from the reference summary and the generated summary. We
examined the correlation between GPT-4 grading with human grading. The results
suggest that this question-answering approach with GPT-4 can be a useful tool
for gauging the quality of the summary.

我们提出了一个新颖的法律摘要评估框架，利用 GPT-4 生成一系列覆盖参考摘要中主要观点和信息的问题 - 回答对。然后，根据参考摘要中的问题，利用 GPT-4 生成回答。最后，GPT-4 对参考摘要和生成摘要的回答进行评分，并检验了与人工评分之间的相关性，结果表明这种基于问题回答的 GPT-4 方法可以成为评估摘要质量的有用工具。