Large Language Models (LLMs) excel in fluency but risk producing inaccurate
content, called "hallucinations." This paper outlines a standardized process
for categorizing fine-grained hallucination types and proposes an innovative
framework--the Progressive Fine-grained Model Editor (PFME)--specifically
designed to detect and correct fine-grained hallucinations in LLMs. PFME
consists of two collaborative modules: the Real-time Fact Retrieval Module and
the Fine-grained Hallucination Detection and Editing Module. The former
identifies key entities in the document and retrieves the latest factual
evidence from credible sources. The latter further segments the document into
sentence-level text and, based on relevant evidence and previously edited
context, identifies, locates, and edits each sentence's hallucination type.
Experimental results on FavaBench and FActScore demonstrate that PFME
outperforms existing methods in fine-grained hallucination detection tasks.
Particularly, when using the Llama3-8B-Instruct model, PFME's performance in
fine-grained hallucination detection with external knowledge assistance
improves by 8.7 percentage points (pp) compared to ChatGPT. In editing tasks,
PFME further enhances the FActScore of FActScore-Alpaca13B and
FActScore-ChatGPT datasets, increasing by 16.2pp and 4.6pp, respectively.

标准化的流程用于分类细粒度幻像类型并提出确定和修正 LLM 中细粒度幻像的创新框架 PFME，由实时事实检索模块和细粒度幻像检测和编辑模块组成，在幻像检测和编辑任务中 PFME 表现出优于现有方法的性能。

PFME：一个用于大规模语言模型的细粒度幻觉检测和编辑的模块化方法

PFME: A Modular Approach for Fine-grained Hallucination Detection and  Editing of Large Language Models

Large language models (LMs) are prone to generate diverse factually incorrect
statements, which are widely called hallucinations. Current approaches
predominantly focus on coarse-grained automatic hallucination detection or
editing, overlooking nuanced error levels. In this paper, we propose a novel
task -- automatic fine-grained hallucination detection -- and present a
comprehensive taxonomy encompassing six hierarchically defined types of
hallucination. To facilitate evaluation, we introduce a new benchmark that
includes fine-grained human judgments on two LM outputs across various domains.
Our analysis reveals that ChatGPT and Llama 2-Chat exhibit hallucinations in
60% and 75% of their outputs, respectively, and a majority of these
hallucinations fall into categories that have been underexplored. As an initial
step to address this, we train FAVA, a retrieval-augmented LM by carefully
designing synthetic data generations to detect and correct fine-grained
hallucinations. On our benchmark, our automatic and human evaluations show that
FAVA significantly outperforms ChatGPT on fine-grained hallucination detection
by a large margin though a large room for future improvement still exists.
FAVA's suggested edits also improve the factuality of LM-generated text,
resulting in 5-10% FActScore improvements.

大语言模型倾向于生成多样的事实不准确的陈述，本文提出了一个新的任务 - 自动细粒度幻觉检测，并提出了一个涵盖六种层次定义的幻觉类型的综合分类法。通过引入一个新的基准测试以评估，我们的分析结果显示 ChatGPT 和 Llama 2-Chat 的输出中有 60% 和 75% 的幻觉，而其中大多数幻觉属于未被充分研究的类别。为了解决这个问题的初步步骤，我们训练了 FAVA，一个通过精心设计的合成数据生成来检测和纠正细粒度幻觉的检索增强的语言模型。在我们的基准测试中，我们的自动和人工评估显示 FAVA 在细粒度幻觉检测方面明显优于 ChatGPT，尽管还存在大量改进的空间。FAVA 提供的修改还提高了语言模型生成文本的准确性，导致了 5-10% 的 FActScore 改进。