Writing radiology reports from medical images requires a high level of domain
expertise. It is time-consuming even for trained radiologists and can be
error-prone for inexperienced radiologists. It would be appealing to automate
this task by leveraging generative AI, which has shown drastic progress in
vision and language understanding. In particular, Large Language Models (LLM)
have demonstrated impressive capabilities recently and continued to set new
state-of-the-art performance on almost all natural language tasks. While many
have proposed architectures to combine vision models with LLMs for multimodal
tasks, few have explored practical fine-tuning strategies. In this work, we
proposed a simple yet effective two-stage fine-tuning protocol to align visual
features to LLM's text embedding space as soft visual prompts. Our framework
with OpenLLaMA-7B achieved state-of-the-art level performance without
domain-specific pretraining. Moreover, we provide detailed analyses of soft
visual prompts and attention mechanisms, shedding light on future research
directions.

利用生成性人工智能进行医学图像的放射学报告撰写可极大地减少了时间成本和错误率。本研究中，我们提出了一个简单而有效的两阶段微调方案，通过软性视觉提示将视觉特征与大型语言模型的文本嵌入空间对齐。我们的框架在没有领域特定预训练的情况下取得了最先进的性能，并对软性视觉提示和注意力机制进行了详细分析，为未来的研究方向提供了启示。