With the development of multimodality and large language models, the deep
learning-based technique for medical image captioning holds the potential to
offer valuable diagnostic recommendations. However, current generic text and
image pre-trained models do not yield satisfactory results when it comes to
describing intricate details within medical images. In this paper, we present a
novel medical image captioning method guided by the segment anything model
(SAM) to enable enhanced encoding with both general and detailed feature
extraction. In addition, our approach employs a distinctive pre-training
strategy with mixed semantic learning to simultaneously capture both the
overall information and finer details within medical images. We demonstrate the
effectiveness of this approach, as it outperforms the pre-trained BLIP2 model
on various evaluation metrics for generating descriptions of medical images.

本文介绍了一种由 SAM 模型引导的新型医学图像字幕方法，以实现对医学图像的总体信息和细节进行增强编码，通过混合语义学习的独特预训练策略，同时捕捉医学图像的整体信息和细微细节，并证明了该方法在生成医学图像描述方面在各种评估指标上优于预训练的 BLIP2 模型。