Brief hospital course (BHC) summaries are common clinical documents generated
by summarizing clinical notes. While large language models (LLMs) depict
remarkable capabilities in automating real-world tasks, their capabilities for
healthcare applications such as BHC synthesis have not been shown. To enable
the adaptation of LLMs for BHC synthesis, we introduce a novel benchmark
consisting of a pre-processed dataset extracted from MIMIC-IV notes,
encapsulating clinical note, and brief hospital course (BHC) pairs. We assess
the performance of two general-purpose LLMs and three healthcare-adapted LLMs
to improve BHC synthesis from clinical notes. Using clinical notes as input for
generating BHCs, we apply prompting-based (using in-context learning) and
fine-tuning-based adaptation strategies to three open-source LLMs
(Clinical-T5-Large, Llama2-13B, FLAN-UL2) and two proprietary LLMs (GPT-3.5,
GPT-4). We quantitatively evaluate the performance of these LLMs across varying
context-length inputs using conventional natural language similarity metrics.
We further perform a qualitative study where five diverse clinicians blindly
compare clinician-written BHCs and two LLM-generated BHCs for 30 samples across
metrics of comprehensiveness, conciseness, factual correctness, and fluency.
Overall, we present a new benchmark and pre-processed dataset for using LLMs in
BHC synthesis from clinical notes. We observe high-quality summarization
performance for both in-context proprietary and fine-tuned open-source LLMs
using both quantitative metrics and a qualitative clinical reader study. We
propose our work as a benchmark to motivate future works to adapt and assess
the performance of LLMs in BHC synthesis.

利用临床备注合成摘要（BHC synthesis）的大型语言模型（LLMs）在临床研究中表现出良好的性能，本研究提出了一个新的评估标准和数据集，用于评估临床备注合成中 LLMs 的性能，并通过定量和定性评估展示了高质量的摘要表现。