Large language models have the potential to be valuable in the healthcare industry, but it's crucial to verify their safety and effectiveness through rigorous evaluation. For this purpose, we comprehensively evaluated both open-source LLMs and Google's new multimodal LLM called Gemini across Medical reasoning, hallucination detection, and Medical Visual Question Answering tasks. While Gemini showed competence, it lagged behind state-of-the-art models like MedPaLM 2 and GPT-4 in diagnostic accuracy. Additionally, Gemini achieved an accuracy of 61.45\% on the medical VQA dataset, significantly lower than GPT-4V's score of 88\%. Our analysis revealed that Gemini is highly susceptible to hallucinations, overconfidence, and knowledge gaps, which indicate risks if deployed uncritically. We also performed a detailed analysis by medical subject and test type, providing actionable feedback for developers and clinicians. To mitigate risks, we applied prompting strategies that improved performance. Additionally, we facilitated future research and development by releasing a Python module for medical LLM evaluation and establishing a dedicated leaderboard on Hugging Face for medical domain LLMs. Python module can be found at https://github.com/promptslab/RosettaEval

对大型语言模型在医疗保健行业的潜在价值进行了全面评估，发现Google的新型多模态模型Gemini在医学推理、幻觉检测和医学视觉问答任务中表现尚可，但在诊断准确性上低于MedPaLM 2和GPT-4等先进模型。分析结果显示Gemini对幻觉、过度自信和知识缺失高度敏感，需要谨慎部署。为提高性能，采用提示策略，并通过发布Python模块和建立医学领域LLM的排行榜促进未来研究与发展。

Gemini在医学院：探索多模态大型语言模型在医学挑战问题和幻觉上的能力