Recent advancements in Large Language Models (LLMs) and Large Multi-modal
Models (LMMs) have shown potential in various medical applications, such as
Intelligent Medical Diagnosis. Although impressive results have been achieved,
we find that existing benchmarks do not reflect the complexity of real medical
reports and specialized in-depth reasoning capabilities. In this work, we
introduced RJUA-MedDQA, a comprehensive benchmark in the field of medical
specialization, which poses several challenges: comprehensively interpreting
imgage content across diverse challenging layouts, possessing numerical
reasoning ability to identify abnormal indicators and demonstrating clinical
reasoning ability to provide statements of disease diagnosis, status and advice
based on medical contexts. We carefully design the data generation pipeline and
proposed the Efficient Structural Restoration Annotation (ESRA) Method, aimed
at restoring textual and tabular content in medical report images. This method
substantially enhances annotation efficiency, doubling the productivity of each
annotator, and yields a 26.8% improvement in accuracy. We conduct extensive
evaluations, including few-shot assessments of 5 LMMs which are capable of
solving Chinese medical QA tasks. To further investigate the limitations and
potential of current LMMs, we conduct comparative experiments on a set of
strong LLMs by using image-text generated by ESRA method. We report the
performance of baselines and offer several observations: (1) The overall
performance of existing LMMs is still limited; however LMMs more robust to
low-quality and diverse-structured images compared to LLMs. (3) Reasoning
across context and image content present significant challenges. We hope this
benchmark helps the community make progress on these challenging tasks in
multi-modal medical document understanding and facilitate its application in
healthcare.

最近对大型语言模型 (LLMs) 和大型多模态模型 (LMMs) 的进展表明其在各种医疗应用中具有潜力，如智能医学诊断。本研究引入了综合医学专业领域的 RJUA-MedDQA 基准测试，用于全面解释各种医学报告的复杂性和专门的深入推理能力。我们设计了数据生成流程，并提出了高效的结构恢复注释 (ESRA) 方法，可显著提高注释效率并提供 26.8% 的准确率改进。通过使用 ESRA 方法生成的图像 - 文本，在 5 个能够解决中文医学问答任务的 LMMs 上进行了全面评估和少样本评估。我们对一组强大的 LLMs 进行了比较实验，发现现有的 LLMs 性能仍受限制，但与 LMMs 相比，LMMs 在低质量和多样化结构的图像上更加稳健。跨上下文和图像内容的推理具有重大挑战。希望此基准测试有助于推动多模态医学文档理解领域的研究并促进其在医疗健康领域的应用。