Inferring over and extracting information from Large Language Models (LLMs)
trained on a large corpus of scientific literature can potentially drive a new
era in biomedical research, reducing the barriers for accessing existing
medical evidence. This work examines the potential of LLMs for dialoguing with
biomedical background knowledge, using the context of antibiotic discovery as
an exemplar motivational scenario. The context of biomedical discovery from
natural products entails understanding the relational evidence between an
organism, an associated chemical and its associated antibiotic properties. We
provide a systematic assessment on the ability of LLMs to encode and express
these relations, verifying for fluency, prompt-alignment, semantic coherence,
factual knowledge and specificity of generated responses. The systematic
analysis is applied to nine state-of-the-art models (including ChatGPT and
GPT-4) in two prompting-based tasks: chemical compound definition generation
and chemical compound-fungus relation determination. Results show that while
recent models have improved in fluency, factual accuracy is still low and
models are biased towards over-represented entities. The ability of LLMs to
serve as biomedical knowledge bases is questioned, and the need for additional
systematic evaluation frameworks is highlighted. The best performing GPT-4
produced a factual definition for 70% of chemical compounds and 43.6% factual
relations to fungi, whereas the best open source model BioGPT-large 30% of the
compounds and 30% of the relations for the best-performing prompt. The results
show that while LLMs are currently not fit for purpose to be used as biomedical
factual knowledge bases, there is a promising emerging property in the
direction of factuality as the models become domain specialised, scale-up in
size and level of human feedback.

本文研究了使用大型语言模型（LLMs）对话生物医学背景知识以及从大量的科学文献中提取信息的潜力，以抗生素发现为例，系统评估了 9 种最先进的模型在生成化合物定义和确定化合物 - 真菌关系方面的能力，并发现虽然最新模型在流畅度方面得到了改善，但其事实准确性仍然很低，而且模型存在偏向于过度呈现某些实体的问题。