Recently, Large Language Models (LLM) have demonstrated impressive capability
to solve a wide range of tasks. However, despite their success across various
tasks, no prior work has investigated their capability in the biomedical domain
yet. To this end, this paper aims to evaluate the performance of LLMs on
benchmark biomedical tasks. For this purpose, we conduct a comprehensive
evaluation of 4 popular LLMs in 6 diverse biomedical tasks across 26 datasets.
To the best of our knowledge, this is the first work that conducts an extensive
evaluation and comparison of various LLMs in the biomedical domain.
Interestingly, we find based on our evaluation that in biomedical datasets that
have smaller training sets, zero-shot LLMs even outperform the current
state-of-the-art fine-tuned biomedical models. This suggests that pretraining
on large text corpora makes LLMs quite specialized even in the biomedical
domain. We also find that not a single LLM can outperform other LLMs in all
tasks, with the performance of different LLMs may vary depending on the task.
While their performance is still quite poor in comparison to the biomedical
models that were fine-tuned on large training sets, our findings demonstrate
that LLMs have the potential to be a valuable tool for various biomedical tasks
that lack large annotated data.

最近，大型语言模型 (LLM) 在解决各种任务方面展现了令人印象深刻的能力。然而，尽管在各种任务中取得了成功，但以前的研究尚未调查它们在生物医学领域的能力。为此，本文旨在评估 LLMs 在基准生物医学任务中的性能。为此，我们对 26 个数据集中 6 个不同生物医学任务的 4 种流行 LLMs 进行了全面评估。据我们所知，这是第一次在生物医学领域对各种 LLMs 进行广泛评估和比较。有趣的是，基于我们的评估结果我们发现，在具有较小训练集的生物医学数据集中，零次矫正的 LLMs 甚至在效果上超过了当前最先进的生物医学模型。这表明，在大型文本语料库上进行预训练使 LLMs 在生物医学领域具有了相当专业的能力。我们还发现，在所有任务中没有单个 LLM 能够胜过其他 LLMs，不同 LLMs 的性能可能会因任务而异。尽管与在大型训练集上进行精细调整的生物医学模型相比，它们的性能仍然相当差，但我们的研究结果表明，LLMs 在缺乏大规模注释数据的各种生物医学任务中具有潜在的价值工具。