The mining of adverse drug events (ADEs) is pivotal in pharmacovigilance,
enhancing patient safety by identifying potential risks associated with
medications, facilitating early detection of adverse events, and guiding
regulatory decision-making. Traditional ADE detection methods are reliable but
slow, not easily adaptable to large-scale operations, and offer limited
information. With the exponential increase in data sources like social media
content, biomedical literature, and Electronic Medical Records (EMR),
extracting relevant ADE-related information from these unstructured texts is
imperative. Previous ADE mining studies have focused on text-based
methodologies, overlooking visual cues, limiting contextual comprehension, and
hindering accurate interpretation. To address this gap, we present a MultiModal
Adverse Drug Event (MMADE) detection dataset, merging ADE-related textual
information with visual aids. Additionally, we introduce a framework that
leverages the capabilities of LLMs and VLMs for ADE detection by generating
detailed descriptions of medical images depicting ADEs, aiding healthcare
professionals in visually identifying adverse events. Using our MMADE dataset,
we showcase the significance of integrating visual cues from images to enhance
overall performance. This approach holds promise for patient safety, ADE
awareness, and healthcare accessibility, paving the way for further exploration
in personalized healthcare.

通过将 ADE 相关的文本信息与视觉辅助工具相结合，我们呈现了一个多模态不良药物事件（MMADE）检测数据集，并引入了一个框架，利用 LLMs 和 VLMs 的能力进行 ADE 检测，生成对描绘 ADE 的医学图像的详细描述，帮助医疗专业人员在视觉上识别不良事件，从而提高总体性能。该方法有望促进患者安全、ADE 意识和医疗可及性，为个性化医疗的进一步探索铺平了道路。