Vision-language models have become increasingly powerful for tasks that
require an understanding of both visual and linguistic elements, bridging the
gap between these modalities. In the context of multimodal clinical AI, there
is a growing need for models that possess domain-specific knowledge, as
existing models often lack the expertise required for medical applications. In
this paper, we take brain abnormalities as an example to demonstrate how to
automatically collect medical image-text aligned data for pretraining from
public resources such as PubMed. In particular, we present a pipeline that
streamlines the pre-training process by initially collecting a large brain
image-text dataset from case reports and published journals and subsequently
constructing a high-performance vision-language model tailored to specific
medical tasks. We also investigate the unique challenge of mapping subfigures
to subcaptions in the medical domain. We evaluated the resulting model with
quantitative and qualitative intrinsic evaluations. The resulting dataset and
our code can be found here
this https URL

本研究旨在展示如何从公共资源如 PubMed 中自动收集医学图像与文本对齐数据，构建用于具体医学任务的高性能视觉语言模型，并解决医学领域中子图到子标题的映射问题。