The foundation models based on pre-training technology have significantly
advanced artificial intelligence from theoretical to practical applications.
These models have facilitated the feasibility of computer-aided diagnosis for
widespread use. Medical contrastive vision-language pre-training, which does
not require human annotations, is an effective approach for guiding
representation learning using description information in diagnostic reports.
However, the effectiveness of pre-training is limited by the large-scale
semantic overlap and shifting problems in medical field. To address these
issues, we propose the Knowledge-Boosting Contrastive Vision-Language
Pre-training framework (KoBo), which integrates clinical knowledge into the
learning of vision-language semantic consistency. The framework uses an
unbiased, open-set sample-wise knowledge representation to measure negative
sample noise and supplement the correspondence between vision-language mutual
information and clinical knowledge. Extensive experiments validate the effect
of our framework on eight tasks including classification, segmentation,
retrieval, and semantic relatedness, achieving comparable or better performance
with the zero-shot or few-shot settings. Our code is open on
this https URL

本文提出了一种知识增强的对比视觉语言预训练（KoBo）框架，该框架将临床知识整合进视觉语言语义一致性的学习中，从而应用于分类、分割、提取和语义相关性的任务中，具有与零样本或少样本设置相当或更好的性能。