The increased use of large language models (LLMs) across a variety of
real-world applications calls for mechanisms to verify the factual accuracy of
their outputs. In this work, we present a holistic end-to-end solution for
annotating the factuality of LLM-generated responses, which encompasses a
multi-stage annotation scheme designed to yield detailed labels concerning the
verifiability and factual inconsistencies found in LLM outputs. We design and
build an annotation tool to speed up the labelling procedure and ease the
workload of raters. It allows flexible incorporation of automatic results in
any stage, e.g. automatically-retrieved evidence. We further construct an
open-domain document-level factuality benchmark in three-level granularity:
claim, sentence and document. Preliminary experiments show that FacTool,
FactScore and Perplexity.ai are struggling to identify false claims with the
best F1=0.53. Annotation tool, benchmark and code are available at
this https URL

该研究介绍了一种针对大型语言模型输出进行事实准确性注释的综合解决方案，包括多阶段的注释方案和注释工具的设计，以识别 LLM 输出中的可验证性和事实不一致性，并构建了三个层次粒度的开放领域文档级事实性基准。初步实验结果表明，已有工具在识别错误声明方面存在困难，最佳 F1=0.53。