Large Language Models (LLMs) are widely used to evaluate natural language
generation tasks as automated metrics. However, the likelihood, a measure of
LLM's plausibility for a sentence, can vary due to superficial differences in
sentences, such as word order and sentence structure. It is therefore possible
that there might be a likelihood bias if LLMs are used for evaluation: they
might overrate sentences with higher likelihoods while underrating those with
lower likelihoods. In this paper, we investigate the presence and impact of
likelihood bias in LLM-based evaluators. We also propose a method to mitigate
the likelihood bias. Our method utilizes highly biased instances as few-shot
examples for in-context learning. Our experiments in evaluating the
data-to-text and grammatical error correction tasks reveal that several LLMs we
test display a likelihood bias. Furthermore, our proposed method successfully
mitigates this bias, also improving evaluation performance (in terms of
correlation of models with human scores) significantly.

大型语言模型广泛用于自然语言生成任务的自动评估指标，然而，由于句子中的表面差异（如词序和句子结构），可能会导致语言模型的可能性有所偏差，本文研究了基于语言模型的评估器中可能存在的可能性偏见，并提出了缓解可能性偏见的方法，该方法利用高度偏倚的实例作为少样本示例进行上下文学习，实验证明我们测试的几个语言模型存在可能性偏见，而且我们提出的方法成功地缓解了这种偏见，并显著提高了评估性能（与人类评分的相关性）。