The emergence of large multimodal models has unlocked remarkable potential in
AI, particularly in pathology. However, the lack of specialized, high-quality
benchmark impeded their development and precise evaluation. To address this, we
introduce PathMMU, the largest and highest-quality expert-validated pathology
benchmark for LMMs. It comprises 33,573 multimodal multi-choice questions and
21,599 images from various sources, and an explanation for the correct answer
accompanies each question. The construction of PathMMU capitalizes on the
robust capabilities of GPT-4V, utilizing approximately 30,000 gathered
image-caption pairs to generate Q\&As. Significantly, to maximize PathMMU's
authority, we invite six pathologists to scrutinize each question under strict
standards in PathMMU's validation and test sets, while simultaneously setting
an expert-level performance benchmark for PathMMU. We conduct extensive
evaluations, including zero-shot assessments of 14 open-sourced and three
closed-sourced LMMs and their robustness to image corruption. We also fine-tune
representative LMMs to assess their adaptability to PathMMU. The empirical
findings indicate that advanced LMMs struggle with the challenging PathMMU
benchmark, with the top-performing LMM, GPT-4V, achieving only a 51.7\%
zero-shot performance, significantly lower than the 71.4\% demonstrated by
human pathologists. After fine-tuning, even open-sourced LMMs can surpass
GPT-4V with a performance of over 60\%, but still fall short of the expertise
shown by pathologists. We hope that the PathMMU will offer valuable insights
and foster the development of more specialized, next-generation LLMs for
pathology.

通过构建专业水准的最大、最高质量的病理学基准测试 PathMMU，我们对大型多模态模型的发展和精确评估进行了研究，发现高级多模态模型在面临挑战性的 PathMMU 基准测试中表现不佳，而且即使进行优化，也无法达到病理学家的专业水准。