Radiology Report Generation (RRG) has achieved significant progress with the
advancements of multimodal generative models. However, the evaluation in the
domain suffers from a lack of fair and robust metrics. We reveal that, high
performance on RRG with existing lexical-based metrics (e.g. BLEU) might be
more of a mirage - a model can get a high BLEU only by learning the template of
reports. This has become an urgent problem for RRG due to the highly
patternized nature of these reports. In this work, we un-intuitively approach
this problem by proposing the Layman's RRG framework, a layman's terms-based
dataset, evaluation and training framework that systematically improves RRG
with day-to-day language. We first contribute the translated Layman's terms
dataset. Building upon the dataset, we then propose a semantics-based
evaluation method, which is proved to mitigate the inflated numbers of BLEU and
provides fairer evaluation. Last, we show that training on the layman's terms
dataset encourages models to focus on the semantics of the reports, as opposed
to overfitting to learning the report templates. We reveal a promising scaling
law between the number of training examples and semantics gain provided by our
dataset, compared to the inverse pattern brought by the original formats. Our
code is available at https://github.com/hegehongcha/LaymanRRG.

通过引入一个以外行人术语为基础的数据集、评估和训练框架来系统地改进放射学报告生成 (RRG)，以解决现有用于评估 RRG 的词汇基准度量方法（如 BLEU）的问题，该方法的高性能可能只是一个幻觉。我们提出了一个基于语义的评估方法，并证明其可以缓解 BLEU 得分的虚高问题，为 RRG 提供更公平的评估。我们还展示了在外行人术语数据集上训练可以促使模型关注报告的语义，而不是过度拟合报告模板。