Reinforcement learning-based large language models, such as ChatGPT, are
believed to have potential to aid human experts in many domains, including
healthcare. There is, however, little work on ChatGPT's ability to perform a
key task in healthcare: formal, probabilistic medical diagnostic reasoning.
This type of reasoning is used, for example, to update a pre-test probability
to a post-test probability. In this work, we probe ChatGPT's ability to perform
this task. In particular, we ask ChatGPT to give examples of how to use Bayes
rule for medical diagnosis. Our prompts range from queries that use terminology
from pure probability (e.g., requests for a "posterior probability") to queries
that use terminology from the medical diagnosis literature (e.g., requests for
a "post-test probability"). We show how the introduction of medical variable
names leads to an increase in the number of errors that ChatGPT makes. Given
our results, we also show how one can use prompt engineering to facilitate
ChatGPT's partial avoidance of these errors. We discuss our results in light of
recent commentaries on sensitivity and specificity. We also discuss how our
results might inform new research directions for large language models.

ChatGPT 通过使用提示工程，展示了在医疗诊断推理中其对贝叶斯规则的运用能力，同时讨论了如何减少错误数量以及对大型语言模型的新研究方向的启示。