Many clinical tasks require an understanding of specialized data, such as
medical images and genomics, which is not typically found in general-purpose
large multimodal models. Building upon Gemini's multimodal models, we develop
several models within the new Med-Gemini family that inherit core capabilities
of Gemini and are optimized for medical use via fine-tuning with 2D and 3D
radiology, histopathology, ophthalmology, dermatology and genomic data.
Med-Gemini-2D sets a new standard for AI-based chest X-ray (CXR) report
generation based on expert evaluation, exceeding previous best results across
two separate datasets by an absolute margin of 1% and 12%, where 57% and 96% of
AI reports on normal cases, and 43% and 65% on abnormal cases, are evaluated as
"equivalent or better" than the original radiologists' reports. We demonstrate
the first ever large multimodal model-based report generation for 3D computed
tomography (CT) volumes using Med-Gemini-3D, with 53% of AI reports considered
clinically acceptable, although additional research is needed to meet expert
radiologist reporting quality. Beyond report generation, Med-Gemini-2D
surpasses the previous best performance in CXR visual question answering (VQA)
and performs well in CXR classification and radiology VQA, exceeding SoTA or
baselines on 17 of 20 tasks. In histopathology, ophthalmology, and dermatology
image classification, Med-Gemini-2D surpasses baselines across 18 out of 20
tasks and approaches task-specific model performance. Beyond imaging,
Med-Gemini-Polygenic outperforms the standard linear polygenic risk score-based
approach for disease risk prediction and generalizes to genetically correlated
diseases for which it has never been trained. Although further development and
evaluation are necessary in the safety-critical medical domain, our results
highlight the potential of Med-Gemini across a wide range of medical tasks.

通过对 Med-Gemini 的多模态模型进行改进，我们在医学领域开发了几个 Med-Gemini 系列模型，通过二维和三维放射学、组织病理学、眼科学、皮肤科学和基因组学数据的微调，继承了 Gemini 的核心能力，并创立了 AI 基于 X 射线的报告生成的新标准，并在多项医学任务中超越了现有基线模型表现。

推进 Gemini 的多模医疗能力

Advancing Multimodal Medical Capabilities of Gemini

Excellence in a wide variety of medical applications poses considerable
challenges for AI, requiring advanced reasoning, access to up-to-date medical
knowledge and understanding of complex multimodal data. Gemini models, with
strong general capabilities in multimodal and long-context reasoning, offer
exciting possibilities in medicine. Building on these core strengths of Gemini,
we introduce Med-Gemini, a family of highly capable multimodal models that are
specialized in medicine with the ability to seamlessly use web search, and that
can be efficiently tailored to novel modalities using custom encoders. We
evaluate Med-Gemini on 14 medical benchmarks, establishing new state-of-the-art
(SoTA) performance on 10 of them, and surpass the GPT-4 model family on every
benchmark where a direct comparison is viable, often by a wide margin. On the
popular MedQA (USMLE) benchmark, our best-performing Med-Gemini model achieves
SoTA performance of 91.1% accuracy, using a novel uncertainty-guided search
strategy. On 7 multimodal benchmarks including NEJM Image Challenges and MMMU
(health & medicine), Med-Gemini improves over GPT-4V by an average relative
margin of 44.5%. We demonstrate the effectiveness of Med-Gemini's long-context
capabilities through SoTA performance on a needle-in-a-haystack retrieval task
from long de-identified health records and medical video question answering,
surpassing prior bespoke methods using only in-context learning. Finally,
Med-Gemini's performance suggests real-world utility by surpassing human
experts on tasks such as medical text summarization, alongside demonstrations
of promising potential for multimodal medical dialogue, medical research and
education. Taken together, our results offer compelling evidence for
Med-Gemini's potential, although further rigorous evaluation will be crucial
before real-world deployment in this safety-critical domain.

Gemini 模型在医学应用领域的卓越表现面临着挑战，要求先进的推理能力、获取最新的医学知识和理解复杂的多模态数据。该研究引入了 Med-Gemini，一类在医学领域具有高度实用性的多模态模型，能够无缝地使用网络搜索，并可以通过定制编码器高效地适应新的模态特征。在 14 个医学基准测试中，Med-Gemini 取得了新的最佳性能，超过了 GPT-4 模型系列，并在多模态基准测试中取得了相对较大的提升。该研究通过突破先前的方法，在长期匿名健康记录和医学视频问答方面展示了 Med-Gemini 的长语境能力和非凡性能，并在医学文本摘要等任务上超越人类专家。尽管在这个安全关键领域中，需要进一步的严格评估，但我们的结果提供了 Med-Gemini 潜力的有力证据。