In recent years, automated radiology report generation has experienced significant growth. This paper introduces MRScore, an automatic evaluation metric tailored for radiology report generation by leveraging Large Language Models (LLMs). Conventional NLG (natural language generation) metrics like BLEU are inadequate for accurately assessing the generated radiology reports, as systematically demonstrated by our observations within this paper. To address this challenge, we collaborated with radiologists to develop a framework that guides LLMs for radiology report evaluation, ensuring alignment with human analysis. Our framework includes two key components: i) utilizing GPT to generate large amounts of training data, i.e., reports with different qualities, and ii) pairing GPT-generated reports as accepted and rejected samples and training LLMs to produce MRScore as the model reward. Our experiments demonstrate MRScore's higher correlation with human judgments and superior performance in model selection compared to traditional metrics. Our code and datasets will be available on GitHub.

该研究介绍了MRScore，一种针对放射学报告生成的自动评估指标，通过利用大型语言模型(LLMs)。相较于传统的自然语言生成(NLG)指标如BLEU，MRScore更准确地评估生成的放射学报告。作者与放射科医生合作开发了一个框架，该框架指导LLMs进行放射学报告评估，以确保与人类分析保持一致。我们的实验证明MRScore与人类判断有更高的相关性，并在模型选择方面表现出优越性能。我们的代码和数据集将在GitHub上提供。

MRScore: 基于LLM的放射学报告生成评估系统