In multi-modal frameworks, the alignment of cross-modal features presents a
significant challenge. The predominant approach in multi-modal pre-training
emphasizes either global or local alignment between modalities, utilizing
extensive datasets. This bottom-up driven method often suffers from a lack of
interpretability, a critical concern in radiology. Previous studies have
integrated high-level labels in medical images or text, but these still rely on
manual annotation, a costly and labor-intensive process. Our work introduces a
novel approach by using eye-gaze data, collected synchronously by radiologists
during diagnostic evaluations. This data, indicating radiologists' focus areas,
naturally links chest X-rays to diagnostic texts. We propose the Eye-gaze
Guided Multi-modal Alignment (EGMA) framework to harness eye-gaze data for
better alignment of image and text features, aiming to reduce reliance on
manual annotations and thus cut training costs. Our model demonstrates robust
performance, outperforming other state-of-the-art methods in zero-shot
classification and retrieval tasks. The incorporation of easily-obtained
eye-gaze data during routine radiological diagnoses signifies a step towards
minimizing manual annotation dependency. Additionally, we explore the impact of
varying amounts of eye-gaze data on model performance, highlighting the
feasibility and utility of integrating this auxiliary data into multi-modal
pre-training.

使用眼动数据来辅助图像和文本特征的对齐，以减少对手动注释的依赖和降低培训成本。同时，探讨了不同量的眼动数据对模型性能的影响，突显将此辅助数据整合到多模态预训练中的可行性和实用性。