Large Language Models (LLMs) have demonstrated remarkable versatility in
recent years, offering potential applications across specialized domains such
as healthcare and medicine. Despite the availability of various open-source
LLMs tailored for health contexts, adapting general-purpose LLMs to the medical
domain presents significant challenges. In this paper, we introduce BioMistral,
an open-source LLM tailored for the biomedical domain, utilizing Mistral as its
foundation model and further pre-trained on PubMed Central. We conduct a
comprehensive evaluation of BioMistral on a benchmark comprising 10 established
medical question-answering (QA) tasks in English. We also explore lightweight
models obtained through quantization and model merging approaches. Our results
demonstrate BioMistral's superior performance compared to existing open-source
medical models and its competitive edge against proprietary counterparts.
Finally, to address the limited availability of data beyond English and to
assess the multilingual generalization of medical LLMs, we automatically
translated and evaluated this benchmark into 7 other languages. This marks the
first large-scale multilingual evaluation of LLMs in the medical domain.
Datasets, multilingual evaluation benchmarks, scripts, and all the models
obtained during our experiments are freely released.

BioMistral 是一个开源的、专门针对生物医学领域的大型语言模型，通过基于 Mistral 模型并在 PubMed Central 上进行进一步预训练来实现。我们在包含 10 个既定医疗问答任务的基准测试上对 BioMistral 进行了全面评估，同时探索了通过量化和模型合并方法获得的轻量级模型。我们的结果表明，与现有的开源医学模型相比，BioMistral 具有出色的性能，并且在与专有模型的竞争中有竞争优势。此外，为了解决英语以外数据的有限可用性，并评估医学领域中多语言的泛化能力，我们将该基准测试自动翻译和评估为其他 7 种语言。这标志着医学领域中大规模多语言评估的首次实现。在我们的实验中获得的数据集、多语言评估基准、脚本以及所有模型都可以免费获取。