We present a radiology-specific multimodal model for the task for generating
radiological reports from chest X-rays (CXRs). Our work builds on the idea that
large language model(s) can be equipped with multimodal capabilities through
alignment with pre-trained vision encoders. On natural images, this has been
shown to allow multimodal models to gain image understanding and description
capabilities. Our proposed model (MAIRA-1) leverages a CXR-specific image
encoder in conjunction with a fine-tuned large language model based on
Vicuna-7B, and text-based data augmentation, to produce reports with
state-of-the-art quality. In particular, MAIRA-1 significantly improves on the
radiologist-aligned RadCliQ metric and across all lexical metrics considered.
Manual review of model outputs demonstrates promising fluency and accuracy of
generated reports while uncovering failure modes not captured by existing
evaluation practices. More information and resources can be found on the
project website: this https URL

我们提出了一种放射学特定的多模态模型，用于从胸部 X 线片 (CXR) 生成放射学报告。我们的工作基于这样一个观点，即大型语言模型可以通过与预训练的视觉编码器对齐来具备多模态能力。在自然图像上，这已被证明可以使多模态模型具有图像理解和描述能力。我们提出的模型 (MAIRA-1) 结合了一个特定于 CXR 的图像编码器和一个基于 Vicuna-7B 的精调大型语言模型，以及基于文本的数据增强方法，以产生具有最先进质量的报告。具体而言，MAIRA-1 在与放射科医生对齐的 RadCliQ 指标和所有考虑的词汇指标上均有显著提高。对模型输出进行手动审核显示出生成报告的流畅性和准确性，同时揭示了现有评估方法未捕捉到的故障模式。更多信息和资源可在项目网站中找到：this https URL。