Models such as GPT-4 and Med-PaLM 2 have demonstrated impressive performance
on a wide variety of biomedical NLP tasks. However, these models have hundreds
of billions of parameters, are computationally expensive to run, require users
to send their input data over the internet, and are trained on unknown data
sources. Can smaller, more targeted models compete? To address this question,
we build and release BioMedLM, a 2.7 billion parameter GPT-style autoregressive
model trained exclusively on PubMed abstracts and full articles. When
fine-tuned, BioMedLM can produce strong multiple-choice biomedical
question-answering results competitive with much larger models, such as
achieving a score of 57.3% on MedMCQA (dev) and 69.0% on the MMLU Medical
Genetics exam. BioMedLM can also be fine-tuned to produce useful answers to
patient questions on medical topics. This demonstrates that smaller models can
potentially serve as transparent, privacy-preserving, economical and
environmentally friendly foundations for particular NLP applications, such as
in biomedicine. The model is available on the Hugging Face Hub:
this https URL

BioMedLM, a 2.7 billion parameter GPT-style autoregressive model trained on PubMed, demonstrates competitive performance in biomedical NLP tasks, highlighting the potential of smaller, targeted models as efficient and environmentally friendly alternatives.

BioMedLM：训练于生物医学文本的 27 亿参数语言模型

BioMedLM: A 2.7B Parameter Language Model Trained On Biomedical Text

Recent artificial intelligence (AI) systems have reached milestones in "grand
challenges" ranging from Go to protein-folding. The capability to retrieve
medical knowledge, reason over it, and answer medical questions comparably to
physicians has long been viewed as one such grand challenge.
Large language models (LLMs) have catalyzed significant progress in medical
question answering; Med-PaLM was the first model to exceed a "passing" score in
US Medical Licensing Examination (USMLE) style questions with a score of 67.2%
on the MedQA dataset. However, this and other prior work suggested significant
room for improvement, especially when models' answers were compared to
clinicians' answers. Here we present Med-PaLM 2, which bridges these gaps by
leveraging a combination of base LLM improvements (PaLM 2), medical domain
finetuning, and prompting strategies including a novel ensemble refinement
approach.
Med-PaLM 2 scored up to 86.5% on the MedQA dataset, improving upon Med-PaLM
by over 19% and setting a new state-of-the-art. We also observed performance
approaching or exceeding state-of-the-art across MedMCQA, PubMedQA, and MMLU
clinical topics datasets.
We performed detailed human evaluations on long-form questions along multiple
axes relevant to clinical applications. In pairwise comparative ranking of 1066
consumer medical questions, physicians preferred Med-PaLM 2 answers to those
produced by physicians on eight of nine axes pertaining to clinical utility (p
< 0.001). We also observed significant improvements compared to Med-PaLM on
every evaluation axis (p < 0.001) on newly introduced datasets of 240 long-form
"adversarial" questions to probe LLM limitations.
While further studies are necessary to validate the efficacy of these models
in real-world settings, these results highlight rapid progress towards
physician-level performance in medical question answering.

Med-PaLM 2 combines improvements in Large Language Models, medical domain fine-tuning, and novel ensemble refinement approaches to achieve a state-of-the-art performance approaching or exceeding physician-level performance in medical question answering.