The advent of Large Language Models (LLMs) has significantly transformed the
AI landscape, enhancing machine learning and AI capabilities. Factuality issue
is a critical concern for LLMs, as they may generate factually incorrect
responses. In this paper, we propose GraphEval to evaluate an LLM's performance
using a substantially large test dataset. Specifically, the test dataset is
retrieved from a large knowledge graph with more than 10 million facts without
expensive human efforts. Unlike conventional methods that evaluate LLMs based
on generated responses, GraphEval streamlines the evaluation process by
creating a judge model to estimate the correctness of the answers given by the
LLM. Our experiments demonstrate that the judge model's factuality assessment
aligns closely with the correctness of the LLM's generated outputs, while also
substantially reducing evaluation costs. Besides, our findings offer valuable
insights into LLM performance across different metrics and highlight the
potential for future improvements in ensuring the factual integrity of LLM
outputs. The code is publicly available at this https URL

提出了 GraphEval 方法使用大规模测试数据集来评估大型语言模型的性能，该方法通过从包含 1000 万多个事实的大型知识图中检索测试数据集来简化评估过程，并创造了一个判断模型来估计语言模型所给出答案的正确性。实验证明判断模型的事实评估与语言模型的生成输出的正确性密切相关，并且大大降低了评估成本，同时还为后续改进确保语言模型输出的事实性提供了有价值的见解。