The increased use of large language models (LLMs) across a variety of
real-world applications calls for mechanisms to verify the factual accuracy of
their outputs. Difficulties lie in assessing the factuality of free-form
responses in open domains. Also, different papers use disparate evaluation
benchmarks and measurements, which renders them hard to compare and hampers
future progress. To mitigate these issues, we propose OpenFactCheck, a unified
factuality evaluation framework for LLMs. OpenFactCheck consists of three
modules: (i) CUSTCHECKER allows users to easily customize an automatic
fact-checker and verify the factual correctness of documents and claims, (ii)
LLMEVAL, a unified evaluation framework assesses LLM's factuality ability from
various perspectives fairly, and (iii) CHECKEREVAL is an extensible solution
for gauging the reliability of automatic fact-checkers' verification results
using human-annotated datasets. OpenFactCheck is publicly released at
this https URL

基于大型语言模型（LLMs）的事实性评估框架 OpenFactCheck，包括自定义自动事实检查器 CUSTCHECKER、多角度评估 LMM 事实性的统一评估框架 LLMEVAL 和使用人工标注数据集评估自动事实检查器验证结果可靠性的 CHECKEREVAL。

OpenFactCheck：面向 LLMs 的事实性评估统一框架

OpenFactCheck: A Unified Framework for Factuality Evaluation of LLMs

Recently, the evaluation of Large Language Models has emerged as a popular
area of research. The three crucial questions for LLM evaluation are ``what,
where, and how to evaluate''. However, the existing research mainly focuses on
the first two questions, which are basically what tasks to give the LLM during
testing and what kind of knowledge it should deal with. As for the third
question, which is about what standards to use, the types of evaluators, how to
score, and how to rank, there hasn't been much discussion. In this paper, we
analyze evaluation methods by comparing various criteria with both manual and
automatic evaluation, utilizing onsite, crowd-sourcing, public annotators and
GPT-4, with different scoring methods and ranking systems. We propose a new
dataset, LLMEval and conduct evaluations on 20 LLMs. A total of 2,186
individuals participated, leading to the generation of 243,337 manual
annotations and 57,511 automatic evaluation results. We perform comparisons and
analyses of different settings and conduct 10 conclusions that can provide some
insights for evaluating LLM in the future. The dataset and the results are
publicly available at this https URL .

最近，大型语言模型的评估已成为一个热门的研究领域。本文分析了不同评估方法，比较了各种标准和评分系统，在多个任务和测试中使用了不同的评估者和评分方法，提出了一个新的数据集 LLMEval，并对 20 个大型语言模型进行了评估，得出了 10 个结论，为将来的语言模型评估提供了一些启示。