Summarizing medical conversations poses unique challenges due to the specialized domain and the difficulty of collecting in-domain training data. In this study, we investigate the performance of state-of-the-art doctor-patient conversation generative summarization models on the out-of-domain data. We divide the summarization model of doctor-patient conversation into two configurations: (1) a general model, without specifying subjective (S), objective (O), and assessment (A) and plan (P) notes; (2) a SOAP-oriented model that generates a summary with SOAP sections. We analyzed the limitations and strengths of the fine-tuning language model-based methods and GPTs on both configurations. We also conducted a Linguistic Inquiry and Word Count analysis to compare the SOAP notes from different datasets. The results exhibit a strong correlation for reference notes across different datasets, indicating that format mismatch (i.e., discrepancies in word distribution) is not the main cause of performance decline on out-of-domain data. Lastly, a detailed analysis of SOAP notes is included to provide insights into missing information and hallucinations introduced by the models.

研究考察了最先进的医患对话生成摘要模型在域外数据上的表现，将医患对话摘要模型分为两种配置，一种是通用模型，另一种是以SOAP部分生成摘要的模型。研究分析了基于微调语言模型和GPT的方法在两种配置上的局限性和优势，以及通过语言查询和词计数分析比较不同数据集中SOAP注释。结果显示，不同数据集中的参考注释之间存在很强的相关性，表明格式不匹配并非域外数据性能下降的主要原因。最后，还进行了SOAP注释的详细分析，以揭示模型引入的信息缺失和产生幻觉的性质。

探究医患对话摘要的稳健性：跨领域SOAP笔记分析