Human evaluation is indispensable and inevitable for assessing the quality of
texts generated by machine learning models or written by humans. However, human
evaluation is very difficult to reproduce and its quality is notoriously
unstable, hindering fair comparisons among different natural language
processing (NLP) models and algorithms. Recently, large language models (LLMs)
have demonstrated exceptional performance on unseen tasks when only the task
instructions are provided. In this paper, we explore if such an ability of the
LLMs can be used as an alternative to human evaluation. We present the LLMs
with the exact same instructions, samples to be evaluated, and questions used
to conduct human evaluation, and then ask the LLMs to generate responses to
those questions; we dub this LLM evaluation. We use human evaluation and LLM
evaluation to evaluate the texts in two NLP tasks: open-ended story generation
and adversarial attacks. We show that the result of LLM evaluation is
consistent with the results obtained by expert human evaluation: the texts
rated higher by human experts are also rated higher by the LLMs. We also find
that the results of LLM evaluation are stable over different formatting of the
task instructions and the sampling algorithm used to generate the answer. We
are the first to show the potential of using LLMs to assess the quality of
texts and discuss the limitations and ethical considerations of LLM evaluation.

本文介绍了使用大型语言模型（LLM）代替人类评估来评估人工智能生成的文本的潜力，探索了 LLM 对两个自然语言处理任务的开放性故事生成和对抗性攻击的评估结果，并发现 LLM 评估结果与人类专家的评估结果保持一致。