The long-standing one-to-many problem of gold standard responses in
open-domain dialogue systems presents challenges for automatic evaluation
metrics. Though prior works have demonstrated some success by applying powerful
Large Language Models (LLMs), existing approaches still struggle with the
one-to-many problem, and exhibit subpar performance in domain-specific
scenarios. We assume the commonsense reasoning biases within LLMs may hinder
their performance in domainspecific evaluations. To address both issues, we
propose a novel framework SLIDE (Small and Large Integrated for Dialogue
Evaluation), that leverages both a small, specialised model (SLM), and LLMs for
the evaluation of open domain dialogues. Our approach introduces several
techniques: (1) Contrastive learning to differentiate between robust and
non-robust response embeddings; (2) A novel metric for semantic sensitivity
that combines embedding cosine distances with similarity learned through neural
networks, and (3) a strategy for incorporating the evaluation results from both
the SLM and LLMs. Our empirical results demonstrate that our approach achieves
state-of-the-art performance in both the classification and evaluation tasks,
and additionally the SLIDE evaluator exhibits better correlation with human
judgements. Our code is available at https://
github.com/hegehongcha/SLIDE-ACL2024.

使用 Small and Large Integrated for Dialogue Evaluation（SLIDE）框架，结合小型专门模型（SLM）和大型语言模型（LLMs）来解决开放领域对话系统中的一对多问题和领域特定场景下的性能不佳问题。该方法通过对比学习、语义敏感度度量和结合 SLM 和 LLMs 的评估结果等技术，实现了分类和评估任务的最新性能，并且 SLIDE 评估器与人工判断更好地相关。