The generative capabilities of LLM models present opportunities in accelerating tasks and concerns with the authenticity of the knowledge it produces. To address the concerns, we present a computational approach that systematically evaluates the factual accuracy of biomedical knowledge that an LLM model has been prompted to generate. Our approach encompasses two processes: the generation of disease-centric associations and the verification of them using the semantic knowledge of the biomedical ontologies. Using ChatGPT as the select LLM model, we designed a set of prompt-engineering processes to generate linkages between diseases, drugs, symptoms, and genes to establish grounds for assessments. Experimental results demonstrate high accuracy in identifying disease terms (88%-97%), drug names (90%-91%), and genetic information (88%-98%). The symptom term identification accuracy was notably lower (49%-61%), as verified against the DOID, ChEBI, SYMPTOM, and GO ontologies accordingly. The verification of associations reveals literature coverage rates of (89%-91%) among disease-drug and disease-gene associations. The low identification accuracy for symptom terms also contributed to the verification of symptom-related associations (49%-62%).

本研究针对大型语言模型生成的生物医学知识的真实性问题，提出了一种系统评估生成内容准确性的计算方法。通过生成疾病相关联想并利用生物医学本体的语义知识进行验证，结果表明在疾病术语、药物名称和遗传信息的识别上具有较高的准确性，而症状术语的识别准确性相对较低。这一研究为检验生物医学知识生成的可靠性提供了新的视角和方法。

从知识生成到知识验证：考察ChatGPT的生物医学生成能力