Many diagnostic errors occur because clinicians cannot easily access relevant
information in patient Electronic Health Records (EHRs). In this work we
propose a method to use LLMs to identify pieces of evidence in patient EHR data
that indicate increased or decreased risk of specific diagnoses; our ultimate
aim is to increase access to evidence and reduce diagnostic errors. In
particular, we propose a Neural Additive Model to make predictions backed by
evidence with individualized risk estimates at time-points where clinicians are
still uncertain, aiming to specifically mitigate delays in diagnosis and errors
stemming from an incomplete differential. To train such a model, it is
necessary to infer temporally fine-grained retrospective labels of eventual
"true" diagnoses. We do so with LLMs, to ensure that the input text is from
before a confident diagnosis can be made. We use an LLM to retrieve an initial
pool of evidence, but then refine this set of evidence according to
correlations learned by the model. We conduct an in-depth evaluation of the
usefulness of our approach by simulating how it might be used by a clinician to
decide between a pre-defined list of differential diagnoses.

使用 LLMs 方法识别患者电子健康记录中表明特定诊断风险增加或减少的证据，以提高证据获取并减少诊断错误。通过神经附加模型在临床医生不确定时点进行预测，具有个体化的风险估计，旨在减少诊断延误和因不完整鉴别引起的错误。使用 LLMs 推断细粒度的过去真实诊断标签，确保输入文本在自信诊断之前。通过模型学习的相关性，从初始证据集中筛选出更精确的证据，通过模拟临床医生在预定义鉴别诊断列表中选择来详细评估我们方法的实用性。

朝着可解释的风险预测减少诊断误差

Towards Reducing Diagnostic Errors with Interpretable Risk Prediction

Background: Large language models (LLMs) such as OpenAI's GPT-4 or Google's
PaLM 2 are proposed as viable diagnostic support tools or even spoken of as
replacements for "curbside consults". However, even LLMs specifically trained
on medical topics may lack sufficient diagnostic accuracy for real-life
applications.
Methods: Using collective intelligence methods and a dataset of 200 clinical
vignettes of real-life cases, we assessed and compared the accuracy of
differential diagnoses obtained by asking individual commercial LLMs (OpenAI
GPT-4, Google PaLM 2, Cohere Command, Meta Llama 2) against the accuracy of
differential diagnoses synthesized by aggregating responses from combinations
of the same LLMs.
Results: We find that aggregating responses from multiple, various LLMs leads
to more accurate differential diagnoses (average accuracy for 3 LLMs:
$75.3\%\pm 1.6pp$) compared to the differential diagnoses produced by single
LLMs (average accuracy for single LLMs: $59.0\%\pm 6.1pp$).
Discussion: The use of collective intelligence methods to synthesize
differential diagnoses combining the responses of different LLMs achieves two
of the necessary steps towards advancing acceptance of LLMs as a diagnostic
support tool: (1) demonstrate high diagnostic accuracy and (2) eliminate
dependence on a single commercial vendor.

通过使用集体智能方法和 200 个真实病例的临床案例数据集，我们评估和比较了使用个别商业大型语言模型（OpenAI GPT-4、Google PaLM 2、Cohere Command、Meta Llama 2）提取的不同诊断与使用组合相同大型语言模型的回答合并后产生的不同诊断之间的准确性，结果发现聚合多个不同大型语言模型的回答能够更准确地产生不同诊断。