Recent advancements in large language models (LLMs) on language modeling and
emergent capabilities make them a promising reference-free evaluator of natural
language generation quality, and a competent alternative to human evaluation.
However, hindered by the closed-source or high computational demand to host and
tune, there is a lack of practice to further calibrate an off-the-shelf
LLM-based evaluator towards better human alignment. In this work, we propose
AutoCalibrate, a multi-stage, gradient-free approach to automatically calibrate
and align an LLM-based evaluator toward human preference. Instead of explicitly
modeling human preferences, we first implicitly encompass them within a set of
human labels. Then, an initial set of scoring criteria is drafted by the
language model itself, leveraging in-context learning on different few-shot
examples. To further calibrate this set of criteria, we select the best
performers and re-draft them with self-refinement. Our experiments on multiple
text quality evaluation datasets illustrate a significant improvement in
correlation with expert evaluation through calibration. Our comprehensive
qualitative analysis conveys insightful intuitions and observations on the
essence of effective scoring criteria.

自动校准基于大语言模型的评估器以提高与人类评估的相关性。

校准基于 LLM 的评估器

Calibrating LLM-Based Evaluator

Automatic dialogue response evaluator has been proposed as an alternative to
automated metrics and human evaluation. However, existing automatic evaluators
achieve only moderate correlation with human judgement and they are not robust.
In this work, we propose to build a reference-free evaluator and exploit the
power of semi-supervised training and pretrained (masked) language models.
Experimental results demonstrate that the proposed evaluator achieves a strong
correlation (> 0.6) with human judgement and generalizes robustly to diverse
responses and corpora. We open-source the code and data in
this https URL

本文提出建立无参考评估器以及利用半监督训练和预训练（掩码）语言模型的方法来改进自动对话响应评估器的性能，实验结果表明所提出的自动评估器与人类判断具有很强的相关性（>0.6）并且能够很好地适应不同的回答和语料库。该研究代码和数据已经开源在指定链接。