Unstructured text in medical notes and dialogues contains rich information.
Recent advancements in Large Language Models (LLMs) have demonstrated superior
performance in question answering and summarization tasks on unstructured text
data, outperforming traditional text analysis approaches. However, there is a
lack of scientific studies in the literature that methodically evaluate and
report on the performance of different LLMs, specifically for domain-specific
data such as medical chart notes. We propose an evaluation approach to analyze
the performance of open-source LLMs such as Llama2 and Mistral for medical
summarization tasks, using GPT-4 as an assessor. Our innovative approach to
quantitative evaluation of LLMs can enable quality control, support the
selection of effective LLMs for specific tasks, and advance knowledge discovery
in digital health.

在医疗笔记和对话中，非结构化文本具有丰富的信息。最近大型语言模型（LLMs）的进展展示了对非结构化文本数据的问题回答和摘要任务的卓越性能，超过了传统的文本分析方法。然而，在文献中缺乏有系统地评估和报告不同 LLMs 性能的科学研究，特别是针对医疗病历等特定领域数据。我们提出了一种评估方法来分析开源 LLMs（如 Llama2 和 Mistral）在医疗摘要任务中的性能，使用 GPT-4 作为评估器。我们创新的 LLMs 定量评估方法可以实现质量控制，支持为特定任务选择有效的 LLMs，并推进数字健康领域的知识发现。

开源语言模型在医学文本数据摘要中的比较分析

Comparative Analysis of Open-Source Language Models in Summarizing  Medical Text Data

Summaries of medical text shall be faithful by being consistent and factual
with source inputs, which is an important but understudied topic for safety and
efficiency in healthcare. In this paper, we investigate and improve
faithfulness in summarization on a broad range of medical summarization tasks.
Our investigation reveals that current summarization models often produce
unfaithful outputs for medical input text. We then introduce FAMESUMM, a
framework to improve faithfulness by fine-tuning pre-trained language models
based on medical knowledge. FAMESUMM performs contrastive learning on designed
sets of faithful and unfaithful summaries, and it incorporates medical terms
and their contexts to encourage faithful generation of medical terms. We
conduct comprehensive experiments on three datasets in two languages: health
question and radiology report summarization datasets in English, and a
patient-doctor dialogue dataset in Chinese. Results demonstrate that FAMESUMM
is flexible and effective by delivering consistent improvements over mainstream
language models such as BART, T5, mT5, and PEGASUS, yielding state-of-the-art
performances on metrics for faithfulness and general quality. Human evaluation
by doctors also shows that FAMESUMM generates more faithful outputs. Our code
is available at https: //github.com/psunlpgroup/FaMeSumm.

我们研究了医学文本概述的忠实性，发现当前的概述模型常常会产生不忠实的输出，因此我们引入了 FAMESUMM 框架来改进忠实性，并通过对医学知识进行预训练模型微调，使用对比学习的方式提高医学术语的忠实生成，实验结果和医生的人工评估都证明了 FAMESUMM 在忠实性和总体质量的指标上具有优越性。