The remarkable capabilities of large language models (LLMs) in language understanding and generation have not rendered them immune to hallucinations. LLMs can still generate plausible-sounding but factually incorrect or fabricated information. As LLM-empowered chatbots become popular, laypeople may frequently ask health-related queries and risk falling victim to these LLM hallucinations, resulting in various societal and healthcare implications. In this work, we conduct a pioneering study of hallucinations in LLM-generated responses to real-world healthcare queries from patients. We propose MedHalu, a carefully crafted first-of-its-kind medical hallucination dataset with a diverse range of health-related topics and the corresponding hallucinated responses from LLMs with labeled hallucination types and hallucinated text spans. We also introduce MedHaluDetect framework to evaluate capabilities of various LLMs in detecting hallucinations. We also employ three groups of evaluators -- medical experts, LLMs, and laypeople -- to study who are more vulnerable to these medical hallucinations. We find that LLMs are much worse than the experts. They also perform no better than laypeople and even worse in few cases in detecting hallucinations. To fill this gap, we propose expert-in-the-loop approach to improve hallucination detection through LLMs by infusing expert reasoning. We observe significant performance gains for all the LLMs with an average macro-F1 improvement of 6.3 percentage points for GPT-4.

本研究针对大型语言模型（LLMs）在医疗相关查询中产生的幻觉现象进行首次深入探索，识别了LLMs在处理健康问题时生成的不真实信息。我们提出了MedHalu，一个创新的医疗幻觉数据集，并引入MedHaluDetect框架评估LLMs检测幻觉的能力，结果表明LLMs在该方面明显不如医学专家。通过引入专家智慧来提升幻觉检测的效果，我们的研究为提高医疗应用领域LLMs的可靠性提供了重要见解。

MedHalu：大型语言模型对医疗查询的幻觉反应