The MEDIQA-M3G 2024 challenge necessitates novel solutions for Multilingual & Multimodal Medical Answer Generation in dermatology (wai Yim et al., 2024a). This paper addresses the limitations of traditional methods by proposing a weakly supervised learning approach for open-ended medical question-answering (QA). Our system leverages readily available MEDIQA-M3G images via a VGG16-CNN-SVM model, enabling multilingual (English, Chinese, Spanish) learning of informative skin condition representations. Using pre-trained QA models, we further bridge the gap between visual and textual information through multimodal fusion. This approach tackles complex, open-ended questions even without predefined answer choices. We empower the generation of comprehensive answers by feeding the ViT-CLIP model with multiple responses alongside images. This work advances medical QA research, paving the way for clinical decision support systems and ultimately improving healthcare delivery.

本研究提出了一种弱监督学习方法，通过使用MEDIQA-M3G图像的VGG16-CNN-SVM模型，并通过多模态融合将视觉和文本信息联系起来，解决了传统方法在开放式医疗问答中的局限性，进一步促进了医学问答研究的发展，为临床决策支持系统的构建提供了新的思路，进而提高了医疗保健的服务水平。

MediFact参加MEDIQA-M3G 2024：多模态学习在皮肤科医学问答中的应用