Hallucinations remain a persistent challenge for LLMs. RAG aims to reduce hallucinations by grounding responses in contexts. However, even when provided context, LLMs still frequently introduce unsupported information or contradictions. This paper presents our efforts to measure LLM hallucinations with a focus on summarization tasks, assessing how often various LLMs introduce hallucinations when summarizing documents. We discuss Vectara's existing LLM hallucination leaderboard, based on the Hughes Hallucination Evaluation Model (HHEM). While HHEM and Vectara's Hallucination Leaderboard have garnered great research interest, we examine challenges faced by HHEM and current hallucination detection methods by analyzing the effectiveness of these methods on existing hallucination datasets. To address these limitations, we propose FaithJudge, an LLM-as-a-judge approach guided by few-shot human hallucination annotations, which substantially improves automated LLM hallucination evaluation over current methods. We introduce an enhanced hallucination leaderboard centered on FaithJudge, alongside our current hallucination leaderboard, enabling more reliable benchmarking of LLMs for hallucinations in RAG.

本文针对LLM在摘要任务中的幻觉问题展开研究，评估LLM在总结文档时引入幻觉的频率。提出了一种新颖的方法FaithJudge，通过少量人类幻觉注释为引导，显著提升了LLM幻觉评估的自动化效果，并建立了以此为中心的改进幻觉排行榜，为LLM在RAG中的性能基准测试提供了更可靠的工具。

利用不断发展的排行榜评估RAG中LLM的可信度