We introduce eCLIP, an enhanced version of the CLIP model that integrates
expert annotations in the form of radiologist eye-gaze heatmaps. It tackles key
challenges in contrastive multi-modal medical imaging analysis, notably data
scarcity and the "modality gap" -- a significant disparity between image and
text embeddings that diminishes the quality of representations and hampers
cross-modal interoperability. eCLIP integrates a heatmap processor and
leverages mixup augmentation to efficiently utilize the scarce expert
annotations, thus boosting the model's learning effectiveness. eCLIP is
designed to be generally applicable to any variant of CLIP without requiring
any modifications of the core architecture. Through detailed evaluations across
several tasks, including zero-shot inference, linear probing, cross-modal
retrieval, and Retrieval Augmented Generation (RAG) of radiology reports using
a frozen Large Language Model, eCLIP showcases consistent improvements in
embedding quality. The outcomes reveal enhanced alignment and uniformity,
affirming eCLIP's capability to harness high-quality annotations for enriched
multi-modal analysis in the medical imaging domain.

eCLIP 是 CLIP 模型的增强版本，通过整合放射科医师眼动热图的专家注释，解决对比性多模式医学图像分析中的关键挑战，特别是数据稀缺和 “模态差异”。它通过整合热图处理器和利用稀缺的专家注释的 mixup 数据增强，提高模型的学习效果。通过多个任务的详细评估，包括零样本推理、线性探测、跨模态检索和使用冻结的大型语言模型进行检索增强生成（RAG）放射学报告，eCLIP 展示了嵌入质量的持续改进，揭示了增强的对齐性和一致性，证实了 eCLIP 在医学图像领域利用高质量注释进行丰富的多模态分析能力。