Excellence in a wide variety of medical applications poses considerable
challenges for AI, requiring advanced reasoning, access to up-to-date medical
knowledge and understanding of complex multimodal data. Gemini models, with
strong general capabilities in multimodal and long-context reasoning, offer
exciting possibilities in medicine. Building on these core strengths of Gemini,
we introduce Med-Gemini, a family of highly capable multimodal models that are
specialized in medicine with the ability to seamlessly use web search, and that
can be efficiently tailored to novel modalities using custom encoders. We
evaluate Med-Gemini on 14 medical benchmarks, establishing new state-of-the-art
(SoTA) performance on 10 of them, and surpass the GPT-4 model family on every
benchmark where a direct comparison is viable, often by a wide margin. On the
popular MedQA (USMLE) benchmark, our best-performing Med-Gemini model achieves
SoTA performance of 91.1% accuracy, using a novel uncertainty-guided search
strategy. On 7 multimodal benchmarks including NEJM Image Challenges and MMMU
(health & medicine), Med-Gemini improves over GPT-4V by an average relative
margin of 44.5%. We demonstrate the effectiveness of Med-Gemini's long-context
capabilities through SoTA performance on a needle-in-a-haystack retrieval task
from long de-identified health records and medical video question answering,
surpassing prior bespoke methods using only in-context learning. Finally,
Med-Gemini's performance suggests real-world utility by surpassing human
experts on tasks such as medical text summarization, alongside demonstrations
of promising potential for multimodal medical dialogue, medical research and
education. Taken together, our results offer compelling evidence for
Med-Gemini's potential, although further rigorous evaluation will be crucial
before real-world deployment in this safety-critical domain.

Gemini 模型在医学应用领域的卓越表现面临着挑战，要求先进的推理能力、获取最新的医学知识和理解复杂的多模态数据。该研究引入了 Med-Gemini，一类在医学领域具有高度实用性的多模态模型，能够无缝地使用网络搜索，并可以通过定制编码器高效地适应新的模态特征。在 14 个医学基准测试中，Med-Gemini 取得了新的最佳性能，超过了 GPT-4 模型系列，并在多模态基准测试中取得了相对较大的提升。该研究通过突破先前的方法，在长期匿名健康记录和医学视频问答方面展示了 Med-Gemini 的长语境能力和非凡性能，并在医学文本摘要等任务上超越人类专家。尽管在这个安全关键领域中，需要进一步的严格评估，但我们的结果提供了 Med-Gemini 潜力的有力证据。

Gemini 模型在医学领域的能力

Capabilities of Gemini Models in Medicine

Deep implicit functions (DIFs) have emerged as a potent and articulate means
of representing 3D shapes. However, methods modeling object categories or
non-rigid entities have mainly focused on single-object scenarios. In this
work, we propose MODIF, a multi-object deep implicit function that jointly
learns the deformation fields and instance-specific latent codes for multiple
objects at once. Our emphasis is on non-rigid, non-interpenetrating entities
such as organs. To effectively capture the interrelation between these entities
and ensure precise, collision-free representations, our approach facilitates
signaling between category-specific fields to adequately rectify shapes. We
also introduce novel inter-object supervision: an attraction-repulsion loss is
formulated to refine contact regions between objects. Our approach is
demonstrated on various medical benchmarks, involving modeling different groups
of intricate anatomical entities. Experimental results illustrate that our
model can proficiently learn the shape representation of each organ and their
relations to others, to the point that shapes missing from unseen instances can
be consistently recovered by our method. Finally, MODIF can also propagate
semantic information throughout the population via accurate point
correspondences

通过多物体深度隐式函数模型（MODIF），我们提出了一种学习多个非刚性实例的变形场和实例特定潜在代码的方法，以实现精确、无碰撞的形状表示，并在医学基准测试中展示了其优越性能。

非刚性物体的跨类别信号隐式建模

Implicit Modeling of Non-rigid Objects with Cross-Category Signals

Generalist foundation models such as GPT-4 have displayed surprising
capabilities in a wide variety of domains and tasks. Yet, there is a prevalent
assumption that they cannot match specialist capabilities of fine-tuned models.
For example, most explorations to date on medical competency benchmarks have
leveraged domain-specific training, as exemplified by efforts on BioGPT and
Med-PaLM. We build on a prior study of GPT-4's capabilities on medical
challenge benchmarks in the absence of special training. Rather than using
simple prompting to highlight the model's out-of-the-box capabilities, we
perform a systematic exploration of prompt engineering. We find that prompting
innovation can unlock deeper specialist capabilities and show that GPT-4 easily
tops prior leading results for medical benchmarks. The prompting methods we
explore are general purpose, and make no specific use of domain expertise,
removing the need for expert-curated content. Our experimental design carefully
controls for overfitting during the prompt engineering process. We introduce
Medprompt, based on a composition of several prompting strategies. With
Medprompt, GPT-4 achieves state-of-the-art results on all nine of the benchmark
datasets in the MultiMedQA suite. The method outperforms leading specialist
models such as Med-PaLM 2 by a significant margin with an order of magnitude
fewer calls to the model. Steering GPT-4 with Medprompt achieves a 27%
reduction in error rate on the MedQA dataset over the best methods to date
achieved with specialist models and surpasses a score of 90% for the first
time. Beyond medical problems, we show the power of Medprompt to generalize to
other domains and provide evidence for the broad applicability of the approach
via studies of the strategy on exams in electrical engineering, machine
learning, philosophy, accounting, law, nursing, and clinical psychology.

通过对 GPT-4 的提示工程进行系统性探索，我们发现启示创新可以释放更深层次的专家能力，并且证明了 GPT-4 在医学基准测试中轻松超越了先前的主要结果。使用 Medprompt，GPT-4 在 MultiMedQA 套件的九个基准数据集上取得了最先进的结果。