Large Language Models (LLMs) have demonstrated remarkable versatility in
recent years, offering potential applications across specialized domains such
as healthcare and medicine. Despite the availability of various open-source
LLMs tailored for health contexts, adapting general-purpose LLMs to the medical
domain presents significant challenges. In this paper, we introduce BioMistral,
an open-source LLM tailored for the biomedical domain, utilizing Mistral as its
foundation model and further pre-trained on PubMed Central. We conduct a
comprehensive evaluation of BioMistral on a benchmark comprising 10 established
medical question-answering (QA) tasks in English. We also explore lightweight
models obtained through quantization and model merging approaches. Our results
demonstrate BioMistral's superior performance compared to existing open-source
medical models and its competitive edge against proprietary counterparts.
Finally, to address the limited availability of data beyond English and to
assess the multilingual generalization of medical LLMs, we automatically
translated and evaluated this benchmark into 7 other languages. This marks the
first large-scale multilingual evaluation of LLMs in the medical domain.
Datasets, multilingual evaluation benchmarks, scripts, and all the models
obtained during our experiments are freely released.

BioMistral 是一个开源的、专门针对生物医学领域的大型语言模型，通过基于 Mistral 模型并在 PubMed Central 上进行进一步预训练来实现。我们在包含 10 个既定医疗问答任务的基准测试上对 BioMistral 进行了全面评估，同时探索了通过量化和模型合并方法获得的轻量级模型。我们的结果表明，与现有的开源医学模型相比，BioMistral 具有出色的性能，并且在与专有模型的竞争中有竞争优势。此外，为了解决英语以外数据的有限可用性，并评估医学领域中多语言的泛化能力，我们将该基准测试自动翻译和评估为其他 7 种语言。这标志着医学领域中大规模多语言评估的首次实现。在我们的实验中获得的数据集、多语言评估基准、脚本以及所有模型都可以免费获取。

BioMistral: 面向医学领域的一组开源的预训练大型语言模型

BioMistral: A Collection of Open-Source Pretrained Large Language Models  for Medical Domains

Pretrained, large, generative language models (LMs) have had great success in
a wide range of sequence tagging and structured prediction tasks. Casting a
sequence tagging task as a Seq2Seq one requires deciding the formats of the
input and output sequences. However, we lack a principled understanding of the
trade-offs associated with these formats (such as the effect on model accuracy,
sequence length, multilingual generalization, hallucination). In this paper, we
rigorously study different formats one could use for casting input text
sentences and their output labels into the input and target (i.e., output) of a
Seq2Seq model. Along the way, we introduce a new format, which we show to to be
both simpler and more effective. Additionally the new format demonstrates
significant gains in the multilingual settings -- both zero-shot transfer
learning and joint training. Lastly, we find that the new format is more robust
and almost completely devoid of hallucination -- an issue we find common in
existing formats. With well over a 1000 experiments studying 14 different
formats, over 7 diverse public benchmarks -- including 3 multilingual datasets
spanning 7 languages -- we believe our findings provide a strong empirical
basis in understanding how we should tackle sequence tagging tasks.

本文通过超过 1000 次实验，研究了 14 种不同的格式，发现其中一个新的格式既简单又有效，在多语言情况下也表现出显著的优势，并且几乎没有幻觉问题。这些发现为我们解决序列标注任务提供了强有力的经验基础。