Recent advances in large foundation models, such as the Segment Anything
Model (SAM), have demonstrated considerable promise across various tasks.
Despite their progress, these models still encounter challenges in specialized
medical image analysis, especially in recognizing subtle inter-class
differences in Diabetic Retinopathy (DR) lesion segmentation. In this paper, we
propose a novel framework that customizes SAM for text-prompted DR lesion
segmentation, termed TP-DRSeg. Our core idea involves exploiting language cues
to inject medical prior knowledge into the vision-only segmentation network,
thereby combining the advantages of different foundation models and enhancing
the credibility of segmentation. Specifically, to unleash the potential of
vision-language models in the recognition of medical concepts, we propose an
explicit prior encoder that transfers implicit medical concepts into explicit
prior knowledge, providing explainable clues to excavate low-level features
associated with lesions. Furthermore, we design a prior-aligned injector to
inject explicit priors into the segmentation process, which can facilitate
knowledge sharing across multi-modality features and allow our framework to be
trained in a parameter-efficient fashion. Experimental results demonstrate the
superiority of our framework over other traditional models and foundation model
variants.

利用语言线索将医学先验知识注入到视觉分割网络中，提出了一种自定义 SAM 用于文本提示的糖尿病视网膜病变分割的新框架 TP-DRSeg，实验结果表明该框架优于传统模型和基础模型变种。