Large Language Models (LLMs) have significantly advanced the field of Natural
Language Processing (NLP), achieving remarkable performance across diverse
tasks and enabling widespread real-world applications. However, LLMs are prone
to hallucination, generating content that either conflicts with established
knowledge or is unfaithful to the original sources. Existing hallucination
benchmarks primarily focus on sentence- or passage-level hallucination
detection, neglecting dialogue-level evaluation, hallucination localization,
and rationale provision. They also predominantly target factuality
hallucinations while underestimating faithfulness hallucinations, often relying
on labor-intensive or non-specialized evaluators. To address these limitations,
we propose HalluDial, the first comprehensive large-scale benchmark for
automatic dialogue-level hallucination evaluation. HalluDial encompasses both
spontaneous and induced hallucination scenarios, covering factuality and
faithfulness hallucinations. The benchmark includes 4,094 dialogues with a
total of 146,856 samples. Leveraging HalluDial, we conduct a comprehensive
meta-evaluation of LLMs' hallucination evaluation capabilities in
information-seeking dialogues and introduce a specialized judge language model,
HalluJudge. The high data quality of HalluDial enables HalluJudge to achieve
superior or competitive performance in hallucination evaluation, facilitating
the automatic assessment of dialogue-level hallucinations in LLMs and providing
valuable insights into this phenomenon. The dataset and the code are available
at this https URL

提出 HalluDial，这是首个用于自动对话级幻觉评估的综合大规模基准测试。HalluDial 包含了以上文提到的内容，并包括了分为自发和感应性的幻觉情景，并涵盖了实际性幻觉和忠实性幻觉。