Large language models (LLMs) have demonstrated powerful text generation
capabilities, bringing unprecedented innovation to the healthcare field. While
LLMs hold immense promise for applications in healthcare, applying them to real
clinical scenarios presents significant challenges, as these models may
generate content that deviates from established medical facts and even exhibit
potential biases. In our research, we develop an augmented LLM framework based
on the Unified Medical Language System (UMLS), aiming to better serve the
healthcare community. We employ LLaMa2-13b-chat and ChatGPT-3.5 as our
benchmark models, and conduct automatic evaluations using the ROUGE Score and
BERTScore on 104 questions from the LiveQA test set. Additionally, we establish
criteria for physician-evaluation based on four dimensions: Factuality,
Completeness, Readability and Relevancy. ChatGPT-3.5 is used for physician
evaluation with 20 questions on the LiveQA test set. Multiple resident
physicians conducted blind reviews to evaluate the generated content, and the
results indicate that this framework effectively enhances the factuality,
completeness, and relevance of generated content. Our research demonstrates the
effectiveness of using UMLS-augmented LLMs and highlights the potential
application value of LLMs in in medical question-answering.

我们的研究开发了一个基于统一医学语言系统（UMLS）的增强型大型语言模型（LLMs）框架，通过自动和医师评估等方法，实验证明了该框架有效地提高了生成内容的准确性、完整性和相关性，同时突出了大型语言模型在医学问答中的潜在应用价值。