In the healthcare domain, summarizing medical questions posed by patients is
critical for improving doctor-patient interactions and medical decision-making.
Although medical data has grown in complexity and quantity, the current body of
research in this domain has primarily concentrated on text-based methods,
overlooking the integration of visual cues. Also prior works in the area of
medical question summarisation have been limited to the English language. This
work introduces the task of multimodal medical question summarization for
codemixed input in a low-resource setting. To address this gap, we introduce
the Multimodal Medical Codemixed Question Summarization MMCQS dataset, which
combines Hindi-English codemixed medical queries with visual aids. This
integration enriches the representation of a patient's medical condition,
providing a more comprehensive perspective. We also propose a framework named
MedSumm that leverages the power of LLMs and VLMs for this task. By utilizing
our MMCQS dataset, we demonstrate the value of integrating visual information
from images to improve the creation of medically detailed summaries. This
multimodal strategy not only improves healthcare decision-making but also
promotes a deeper comprehension of patient queries, paving the way for future
exploration in personalized and responsive medical care. Our dataset, code, and
pre-trained models will be made publicly available.

在医疗卫生领域，对患者所提出的医学问题进行概括对于改善医患交流及医疗决策至关重要。然而，当前该领域的研究主要集中在基于文本的方法上，忽视了视觉线索的整合。这项工作在资源匮乏的环境中引入了多模态医学问题概括的任务，针对印度 - 英语代码混合的医学查询，结合了视觉辅助信息。该整合丰富了患者医疗状况的表达，提供了更全面的视角。我们还提出了一个名为 MedSumm 的框架，利用语言模型和视觉模型的力量来完成这项任务。通过使用我们的 MMCQS 数据集，我们展示了利用图像中的视觉信息来改进医学详细摘要的价值。这种多模态策略不仅改善了医疗决策，还促进了对患者问题的更深入理解，为个性化和响应式医疗护理的未来探索铺平了道路。我们的数据集、代码和预训练模型将会公开提供。

MedSumm: 一种多模态方法来对混编印地语 - 英语临床查询进行摘要

MedSumm: A Multimodal Approach to Summarizing Code-Mixed Hindi-English  Clinical Queries

In the era of modern healthcare, swiftly generating medical question
summaries is crucial for informed and timely patient care. Despite the
increasing complexity and volume of medical data, existing studies have focused
solely on text-based summarization, neglecting the integration of visual
information. Recognizing the untapped potential of combining textual queries
with visual representations of medical conditions, we introduce the Multimodal
Medical Question Summarization (MMQS) Dataset. This dataset, a major
contribution to our work, pairs medical queries with visual aids, facilitating
a richer and more nuanced understanding of patient needs. We also propose a
framework, utilizing the power of Contrastive Language Image Pretraining(CLIP)
and Large Language Models(LLMs), consisting of four modules that identify
medical disorders, generate relevant context, filter medical concepts, and
craft visually aware summaries. Our comprehensive framework harnesses the power
of CLIP, a multimodal foundation model, and various general-purpose LLMs,
comprising four main modules: the medical disorder identification module, the
relevant context generation module, the context filtration module for
distilling relevant medical concepts and knowledge, and finally, a
general-purpose LLM to generate visually aware medical question summaries.
Leveraging our MMQS dataset, we showcase how visual cues from images enhance
the generation of medically nuanced summaries. This multimodal approach not
only enhances the decision-making process in healthcare but also fosters a more
nuanced understanding of patient queries, laying the groundwork for future
research in personalized and responsive medical care

在现代医疗时代，迅速生成医疗问题摘要对知情和及时的患者护理至关重要。本文介绍了多模态医疗问题摘要（MMQS）数据集，该数据集将医疗查询与图像辅助相结合，便于更丰富、更细致地理解患者需求。我们提出了一个基于 Contrastive Language Image Pretraining (CLIP) 和 Large Language Models (LLMs) 的框架，包括四个模块，用于识别医疗障碍、生成相关上下文、过滤医疗概念和制作具有视觉感知的摘要。通过利用我们的 MMQS 数据集，展示了图像视觉线索如何增强医学细致摘要的生成。这种多模态方法不仅提升了医疗决策过程，还促进了对患者查询的更细致理解，为个性化和响应式医疗护理的未来研究奠定了基础。