With the development of multimodality and large language models, the deep
learning-based technique for medical image captioning holds the potential to
offer valuable diagnostic recommendations. However, current generic text and
image pre-trained models do not yield satisfactory results when it comes to
describing intricate details within medical images. In this paper, we present a
novel medical image captioning method guided by the segment anything model
(SAM) to enable enhanced encoding with both general and detailed feature
extraction. In addition, our approach employs a distinctive pre-training
strategy with mixed semantic learning to simultaneously capture both the
overall information and finer details within medical images. We demonstrate the
effectiveness of this approach, as it outperforms the pre-trained BLIP2 model
on various evaluation metrics for generating descriptions of medical images.

本文介绍了一种由 SAM 模型引导的新型医学图像字幕方法，以实现对医学图像的总体信息和细节进行增强编码，通过混合语义学习的独特预训练策略，同时捕捉医学图像的整体信息和细微细节，并证明了该方法在生成医学图像描述方面在各种评估指标上优于预训练的 BLIP2 模型。

基于 Sam 引导的增强细粒度混合语义学习的医学图像字幕生成

Sam-Guided Enhanced Fine-Grained Encoding with Mixed Semantic Learning  for Medical Image Captioning

Automated radiology report generation aims to generate radiology reports that
contain rich, fine-grained descriptions of radiology imaging. Compared with
image captioning in the natural image domain, medical images are very similar
to each other, with only minor differences in the occurrence of diseases. Given
the importance of these minor differences in the radiology report, it is
crucial to encourage the model to focus more on the subtle regions of disease
occurrence. Secondly, the problem of visual and textual data biases is serious.
Not only do normal cases make up the majority of the dataset, but sentences
describing areas with pathological changes also constitute only a small part of
the paragraph. Lastly, generating medical image reports involves the challenge
of long text generation, which requires more expertise and empirical training
in medical knowledge. As a result, the difficulty of generating such reports is
increased. To address these challenges, we propose a disease-oriented retrieval
framework that utilizes similar reports as prior knowledge references. We
design a factual consistency captioning generator to generate more accurate and
factually consistent disease descriptions. Our framework can find most similar
reports for a given disease from the CXR database by retrieving a
disease-oriented mask consisting of the position and morphological
characteristics. By referencing the disease-oriented similar report and the
visual features, the factual consistency model can generate a more accurate
radiology report.

提出了一种利用疾病定向检索框架以及事实一致性字幕生成器解决医学图像报告自动生成中的疾病区域强调、视觉和文本数据偏见、长文本生成等难题的方法。

像放射科医师一样阅读放射学影像

Reading Radiology Imaging Like The Radiologist

The image captioning task is increasingly prevalent in artificial
intelligence applications for medicine. One important application is clinical
report generation from chest radiographs. The clinical writing of unstructured
reports is time consuming and error-prone. An automated system would improve
standardization, error reduction, time consumption, and medical accessibility.
In this paper we demonstrate the importance of domain specific pre-training and
propose a modified transformer architecture for the medical image captioning
task. To accomplish this, we train a series of modified transformers to
generate clinical reports from chest radiograph image input. These modified
transformers include: a meshed-memory augmented transformer architecture with
visual extractor using ImageNet pre-trained weights, a meshed-memory augmented
transformer architecture with visual extractor using CheXpert pre-trained
weights, and a meshed-memory augmented transformer whose encoder is passed the
concatenated embeddings using both ImageNet pre-trained weights and CheXpert
pre-trained weights. We use BLEU(1-4), ROUGE-L, CIDEr, and the clinical
CheXbert F1 scores to validate our models and demonstrate competitive scores
with state of the art models. We provide evidence that ImageNet pre-training is
ill-suited for the medical image captioning task, especially for less frequent
conditions (eg: enlarged cardiomediastinum, lung lesion, pneumothorax).
Furthermore, we demonstrate that the double feature model improves performance
for specific medical conditions (edema, consolidation, pneumothorax, support
devices) and overall CheXbert F1 score, and should be further developed in
future work. Such a double feature model, including both ImageNet pre-training
as well as domain specific pre-training, could be used in a wide range of image
captioning models in medicine.

本论文提出了通过域特定预训练和提出改进的 transformer 架构从胸部放射状图像输入生成临床报告的方法，该方法具有标准化、减少错误、节省时间和便于医学访问的优点，同时还使用验证数据集的指标验证了其性能。