Writing radiology reports from medical images requires a high level of domain expertise. It is time-consuming even for trained radiologists and can be error-prone for inexperienced radiologists. It would be appealing to automate this task by leveraging generative AI, which has shown drastic progress in vision and language understanding. In particular, Large Language Models (LLM) have demonstrated impressive capabilities recently and continued to set new state-of-the-art performance on almost all natural language tasks. While many have proposed architectures to combine vision models with LLMs for multimodal tasks, few have explored practical fine-tuning strategies. In this work, we proposed a simple yet effective two-stage fine-tuning protocol to align visual features to LLM's text embedding space as soft visual prompts. Our framework with OpenLLaMA-7B achieved state-of-the-art level performance without domain-specific pretraining. Moreover, we provide detailed analyses of soft visual prompts and attention mechanisms, shedding light on future research directions.

利用生成性人工智能进行医学图像的放射学报告撰写可极大地减少了时间成本和错误率。本研究中，我们提出了一个简单而有效的两阶段微调方案，通过软性视觉提示将视觉特征与大型语言模型的文本嵌入空间对齐。我们的框架在没有领域特定预训练的情况下取得了最先进的性能，并对软性视觉提示和注意力机制进行了详细分析，为未来的研究方向提供了启示。

有效微调以提升大型多模态放射学报告生成模型