We propose a new automated evaluation metric for machine-generated radiology
reports using the successful COMET architecture adapted for the radiology
domain. We train and publish four medically-oriented model checkpoints,
including one trained on RadGraph, a radiology knowledge graph. Our results
show that our metric correlates moderately to high with established metrics
such as BERTscore, BLEU, and CheXbert scores. Furthermore, we demonstrate that
one of our checkpoints exhibits a high correlation with human judgment, as
assessed using the publicly available annotations of six board-certified
radiologists, using a set of 200 reports. We also performed our own analysis
gathering annotations with two radiologists on a collection of 100 reports. The
results indicate the potential effectiveness of our method as a
radiology-specific evaluation metric. The code, data, and model checkpoints to
reproduce our findings will be publicly available.

我们提出了一种适用于放射学领域的新的自动评估度量标准，使用成功的 COMET 架构。通过在放射学知识图谱 RadGraph 上训练和发布四个面向医学的模型检查点，我们的结果表明我们的度量标准与 BERTscore、BLEU 和 CheXbert 分数等已有度量标准呈现中高度相关性。此外，我们证明了我们的一个检查点与使用公开可用的六个董事认证的放射科医生的注释集评估人类判断具有很高的相关性，并使用一套包含 200 份报告的数据集进行了分析，同时还与两名放射科医生在一组包含 100 份报告的集合上进行了自己的分析。结果表明我们的方法有潜在效力作为放射学特定评估度量标准。我们的代码、数据和模型检查点将公开提供。