This research focuses on evaluating the non-commercial open-source large
language models (LLMs) Meditron, MedAlpaca, Mistral, and Llama-2 for their
efficacy in interpreting medical guidelines saved in PDF format. As a specific
test scenario, we applied these models to the guidelines for hypertension in
children and adolescents provided by the European Society of Cardiology (ESC).
Leveraging Streamlit, a Python library, we developed a user-friendly medical
document chatbot tool (MedDoc-Bot). This tool enables authorized users to
upload PDF files and pose questions, generating interpretive responses from
four locally stored LLMs. A pediatric expert provides a benchmark for
evaluation by formulating questions and responses extracted from the ESC
guidelines. The expert rates the model-generated responses based on their
fidelity and relevance. Additionally, we evaluated the METEOR and chrF metric
scores to assess the similarity of model responses to reference answers. Our
study found that Llama-2 and Mistral performed well in metrics evaluation.
However, Llama-2 was slower when dealing with text and tabular data. In our
human evaluation, we observed that responses created by Mistral, Meditron, and
Llama-2 exhibited reasonable fidelity and relevance. This study provides
valuable insights into the strengths and limitations of LLMs for future
developments in medical document interpretation. Open-Source Code:
this https URL

这项研究评估了非商业开源大型语言模型（LLMs）Meditron，MedAlpaca，Mistral 和 Llama-2 在解释保存为 PDF 格式的医学指南方面的效力。研究开发了一个用户友好的医学文档聊天机器人工具（MedDoc-Bot），能够上传 PDF 文件并提出问题，从四个本地存储的 LLM 中生成解释性回复。研究发现，Llama-2 和 Mistral 在度量评估中表现良好，但 Llama-2 在处理文本和表格数据时较慢。