Background: Large language models (LLMs) such as OpenAI's GPT-4 or Google's
PaLM 2 are proposed as viable diagnostic support tools or even spoken of as
replacements for "curbside consults". However, even LLMs specifically trained
on medical topics may lack sufficient diagnostic accuracy for real-life
applications.
Methods: Using collective intelligence methods and a dataset of 200 clinical
vignettes of real-life cases, we assessed and compared the accuracy of
differential diagnoses obtained by asking individual commercial LLMs (OpenAI
GPT-4, Google PaLM 2, Cohere Command, Meta Llama 2) against the accuracy of
differential diagnoses synthesized by aggregating responses from combinations
of the same LLMs.
Results: We find that aggregating responses from multiple, various LLMs leads
to more accurate differential diagnoses (average accuracy for 3 LLMs:
$75.3\%\pm 1.6pp$) compared to the differential diagnoses produced by single
LLMs (average accuracy for single LLMs: $59.0\%\pm 6.1pp$).
Discussion: The use of collective intelligence methods to synthesize
differential diagnoses combining the responses of different LLMs achieves two
of the necessary steps towards advancing acceptance of LLMs as a diagnostic
support tool: (1) demonstrate high diagnostic accuracy and (2) eliminate
dependence on a single commercial vendor.

通过使用集体智能方法和 200 个真实病例的临床案例数据集，我们评估和比较了使用个别商业大型语言模型（OpenAI GPT-4、Google PaLM 2、Cohere Command、Meta Llama 2）提取的不同诊断与使用组合相同大型语言模型的回答合并后产生的不同诊断之间的准确性，结果发现聚合多个不同大型语言模型的回答能够更准确地产生不同诊断。