Automated retinal image medical description generation is crucial for
streamlining medical diagnosis and treatment planning. Existing challenges
include the reliance on learned retinal image representations, difficulties in
handling multiple imaging modalities, and the lack of clinical context in
visual representations. Addressing these issues, we propose the Multi-Modal
Medical Transformer (M3T), a novel deep learning architecture that integrates
visual representations with diagnostic keywords. Unlike previous studies
focusing on specific aspects, our approach efficiently learns contextual
information and semantics from both modalities, enabling the generation of
precise and coherent medical descriptions for retinal images. Experimental
studies on the DeepEyeNet dataset validate the success of M3T in meeting
ophthalmologists' standards, demonstrating a substantial 13.5% improvement in
BLEU@4 over the best-performing baseline model.

提出了一种集成视觉表示和诊断关键词的新型深度学习架构 (Multi-Modal Medical Transformer, M3T)，用于眼底图像的生成精确和连贯的医学描述。在 DeepEyeNet 数据集上的实验研究验证了 M3T 在满足眼科医生标准方面的成功，相比最佳基线模型，BLEU@4 有了显著的 13.5% 改进。