Radiology Report Generation (R2Gen) demonstrates how Multi-modal Large
Language Models (MLLMs) can automate the creation of accurate and coherent
radiological reports. Existing methods often hallucinate details in text-based
reports that don't accurately reflect the image content. To mitigate this, we
introduce a novel strategy, SERPENT-VLM (SElf Refining Radiology RePort
GENeraTion using Vision Language Models), which improves the R2Gen task by
integrating a self-refining mechanism into the MLLM framework. We employ a
unique self-supervised loss that leverages similarity between pooled image
representations and the contextual representations of the generated
radiological text, alongside the standard Causal Language Modeling objective,
to refine image-text representations. This allows the model to scrutinize and
align the generated text through dynamic interaction between a given image and
the generated text, therefore reducing hallucination and continuously enhancing
nuanced report generation. SERPENT-VLM outperforms existing baselines such as
LLaVA-Med, BiomedGPT, etc., achieving SoTA performance on the IU X-ray and
Radiology Objects in COntext (ROCO) datasets, and also proves to be robust
against noisy images. A qualitative case study emphasizes the significant
advancements towards more sophisticated MLLM frameworks for R2Gen, opening
paths for further research into self-supervised refinement in the medical
imaging domain.

尽管现有方法经常会在基于文本的报告中凭空想象细节，而不能准确反映图像内容，但《Radiology Report Generation（R2Gen）》展示了多模态大型语言模型（MLLMs）如何自动化生成准确和连贯的放射学报告。为了缓解这个问题，我们引入了一种新颖的策略 SERPENT-VLM，将自我完善的机制集成到 MLLM 框架中，通过利用生成的放射学文本的上下文表示和汇总图像表示之间的相似性，在标准因果语言建模目标的基础上采用独特的自监督损失来完善图像 - 文本表示，从而使模型能够通过给定图像和生成文本之间的动态交互来审视和对齐生成的文本，从而减少幻觉并持续增强细微的报告生成能力。SERPENT-VLM 在 IU X-ray 和 Radiology Objects in COntext（ROCO）数据集上优于现有的基线方法，如 LLaVA-Med、BiomedGPT 等，并且证明在嘈杂的图像环境中具有稳健性。定性案例研究强调了 MLLM 框架在 R2Gen 中向更复杂方向发展的重要进展，为医学成像领域的自监督完善研究打开了新的研究路径。