We evaluate four state-of-the-art instruction-tuned large language models (LLMs) -- ChatGPT, Flan-T5 UL2, Tk-Instruct, and Alpaca -- on a set of 13 real-world clinical and biomedical natural language processing (NLP) tasks in English, such as named-entity recognition (NER), question-answering (QA), relation extraction (RE), etc. Our overall results demonstrate that the evaluated LLMs begin to approach performance of state-of-the-art models in zero- and few-shot scenarios for most tasks, and particularly well for the QA task, even though they have never seen examples from these tasks before. However, we observed that the classification and RE tasks perform below what can be achieved with a specifically trained model for the medical field, such as PubMedBERT. Finally, we noted that no LLM outperforms all the others on all the studied tasks, with some models being better suited for certain tasks than others.

评估四个最先进的面向指令的大型语言模型（ChatGPT、Flan-T5 UL2、Tk-Instruct和Alpaca）在13项真实世界的临床和生物医学自然语言处理（NLP）任务，如命名实体识别（NER）、问答（QA）、关系抽取（RE）等方面的表现。总体结果表明，评估的语言模型在大多数任务的零样本和少样本场景中已经接近最先进模型的性能，尤其在QA任务中表现出色，即使它们之前从未见过这些任务的示例。然而，我们观察到分类和RE任务的性能低于专门训练用于医学领域的模型（如PubMedBERT）所能达到的水平。最后，我们注意到没有一个语言模型在所有研究任务中都胜过其他模型，某些模型在特定任务中更适合。

应用于临床和生物医学任务的指导微调大型语言模型的零样本和少样本研究