Existing reference-free turn-level evaluation metrics for chatbots
inadequately capture the interaction between the user and the system.
Consequently, they often correlate poorly with human evaluations. To address
this issue, we propose a novel model-agnostic approach that leverages
Conditional Pointwise Mutual Information (C-PMI) to measure the turn-level
interaction between the system and the user based on a given evaluation
dimension. Experimental results on the widely used FED dialogue evaluation
dataset demonstrate that our approach significantly improves the correlation
with human judgment compared with existing evaluation systems. By replacing the
negative log-likelihood-based scorer with our proposed C-PMI scorer, we achieve
a relative 60.5% higher Spearman correlation on average for the FED evaluation
metric. Our code is publicly available at this https URL

提出了一种新的模型无关方法，利用条件点互信息来衡量给定评估维度下系统和用户之间的对话交互，实验结果显示，与现有的评估系统相比，该方法在广泛使用的 FED 对话评估数据集上显著提高了与人类判断的相关性，在 FED 评估指标上平均达到了 60.5％的相关性提高率。