In recent years, automated radiology report generation has experienced
significant growth. This paper introduces MRScore, an automatic evaluation
metric tailored for radiology report generation by leveraging Large Language
Models (LLMs). Conventional NLG (natural language generation) metrics like BLEU
are inadequate for accurately assessing the generated radiology reports, as
systematically demonstrated by our observations within this paper. To address
this challenge, we collaborated with radiologists to develop a framework that
guides LLMs for radiology report evaluation, ensuring alignment with human
analysis. Our framework includes two key components: i) utilizing GPT to
generate large amounts of training data, i.e., reports with different
qualities, and ii) pairing GPT-generated reports as accepted and rejected
samples and training LLMs to produce MRScore as the model reward. Our
experiments demonstrate MRScore's higher correlation with human judgments and
superior performance in model selection compared to traditional metrics. Our
code and datasets will be available on GitHub.

该研究介绍了 MRScore，一种针对放射学报告生成的自动评估指标，通过利用大型语言模型 (LLMs)。相较于传统的自然语言生成 (NLG) 指标如 BLEU，MRScore 更准确地评估生成的放射学报告。作者与放射科医生合作开发了一个框架，该框架指导 LLMs 进行放射学报告评估，以确保与人类分析保持一致。我们的实验证明 MRScore 与人类判断有更高的相关性，并在模型选择方面表现出优越性能。我们的代码和数据集将在 GitHub 上提供。