The evaluation of natural language generation (NLG) tasks is a significant
and longstanding research issue. With the recent emergence of powerful large
language models (LLMs), some studies have turned to LLM-based automatic
evaluation methods, which demonstrate great potential to become a new
evaluation paradigm following traditional string-based and model-based metrics.
However, despite the improved performance of existing methods, they still
possess some deficiencies, such as dependency on references and limited
evaluation flexibility. Therefore, in this paper, we meticulously construct a
large-scale NLG evaluation corpus NLG-Eval with human and GPT-4 annotations to
alleviate the lack of relevant data in this field. Furthermore, we propose
Themis, an LLM dedicated to NLG evaluation, which has been trained with our
designed multi-perspective consistency and rating-oriented preference alignment
methods. Themis can conduct flexible and interpretable evaluations without
references, and it exhibits superior evaluation performance on various NLG
tasks, simultaneously generalizing well to unseen tasks and surpassing other
evaluation models, including GPT-4.

为了解决自然语言生成任务（NLG）的评估问题，本文构建了一个大规模的 NLG 评估语料库 NLG-Eval，并提出了一个专门用于 NLG 评估的大型语言模型 Themis，该模型通过多角度一致性和基于评分偏好的方法进行训练，能够灵活、可解释地进行评估，无需参考，并在各种 NLG 任务上展现出优越的评估性能，同时在未见任务上具有良好的泛化能力，超越了其他评估模型，包括 GPT-4。