Large Language Models (LLMs), such as ChatGPT/GPT-4, have garnered widespread
attention owing to their myriad of practical applications, yet their adoption
has been constrained by issues of fact-conflicting hallucinations across web
platforms. The assessment of factuality in text, produced by LLMs, remains
inadequately explored, extending not only to the judgment of vanilla facts but
also encompassing the evaluation of factual errors emerging in complex
inferential tasks like multi-hop, and etc. In response, we introduce FactCHD, a
fact-conflicting hallucination detection benchmark meticulously designed for
LLMs. Functioning as a pivotal tool in evaluating factuality within
"Query-Respons" contexts, our benchmark assimilates a large-scale dataset,
encapsulating a broad spectrum of factuality patterns, such as vanilla,
multi-hops, comparison, and set-operation patterns. A distinctive feature of
our benchmark is its incorporation of fact-based chains of evidence, thereby
facilitating comprehensive and conducive factual reasoning throughout the
assessment process. We evaluate multiple LLMs, demonstrating the effectiveness
of the benchmark and current methods fall short of faithfully detecting factual
errors. Furthermore, we present TRUTH-TRIANGULATOR that synthesizes reflective
considerations by tool-enhanced ChatGPT and LoRA-tuning based on Llama2, aiming
to yield more credible detection through the amalgamation of predictive results
and evidence. The benchmark dataset and source code will be made available in
this https URL

通过 FactCHD 基准测试，评估 Large Language Models 中事实性的判断，在检测虚假信息方面取得了有效成果，且当前方法在准确检测事实错误方面存在不足。