We introduce eCLIP, an enhanced version of the CLIP model that integrates
expert annotations in the form of radiologist eye-gaze heatmaps. It tackles key
challenges in contrastive multi-modal medical imaging analysis, notably data
scarcity and the "modality gap" -- a significant disparity between image and
text embeddings that diminishes the quality of representations and hampers
cross-modal interoperability. eCLIP integrates a heatmap processor and
leverages mixup augmentation to efficiently utilize the scarce expert
annotations, thus boosting the model's learning effectiveness. eCLIP is
designed to be generally applicable to any variant of CLIP without requiring
any modifications of the core architecture. Through detailed evaluations across
several tasks, including zero-shot inference, linear probing, cross-modal
retrieval, and Retrieval Augmented Generation (RAG) of radiology reports using
a frozen Large Language Model, eCLIP showcases consistent improvements in
embedding quality. The outcomes reveal enhanced alignment and uniformity,
affirming eCLIP's capability to harness high-quality annotations for enriched
multi-modal analysis in the medical imaging domain.

eCLIP 是 CLIP 模型的增强版本，通过整合放射科医师眼动热图的专家注释，解决对比性多模式医学图像分析中的关键挑战，特别是数据稀缺和 “模态差异”。它通过整合热图处理器和利用稀缺的专家注释的 mixup 数据增强，提高模型的学习效果。通过多个任务的详细评估，包括零样本推理、线性探测、跨模态检索和使用冻结的大型语言模型进行检索增强生成（RAG）放射学报告，eCLIP 展示了嵌入质量的持续改进，揭示了增强的对齐性和一致性，证实了 eCLIP 在医学图像领域利用高质量注释进行丰富的多模态分析能力。

优化医学多模态对比学习：专家批注

Improving Medical Multi-modal Contrastive Learning with Expert  Annotations

This paper aims to establish a generic multi-modal foundation model that has
the scalable capability to massive downstream applications in E-commerce.
Recently, large-scale vision-language pretraining approaches have achieved
remarkable advances in the general domain. However, due to the significant
differences between natural and product images, directly applying these
frameworks for modeling image-level representations to E-commerce will be
inevitably sub-optimal. To this end, we propose an instance-centric multi-modal
pretraining paradigm called ECLIP in this work. In detail, we craft a decoder
architecture that introduces a set of learnable instance queries to explicitly
aggregate instance-level semantics. Moreover, to enable the model to focus on
the desired product instance without reliance on expensive manual annotations,
two specially configured pretext tasks are further proposed. Pretrained on the
100 million E-commerce-related data, ECLIP successfully extracts more generic,
semantic-rich, and robust representations. Extensive experimental results show
that, without further fine-tuning, ECLIP surpasses existing methods by a large
margin on a broad range of downstream tasks, demonstrating the strong
transferability to real-world E-commerce applications.

本研究提出了一种实例为中心的多模态预训练范式 ECLIP，通过引入可学习实例查询的解码器架构和两个预处理任务，从 1 亿个电子商务相关数据中预训练出语义丰富、稳健的模型，进而在广泛的下游任务中超越现有方法，证明其在现实世界中的广泛可转移性。