Automatic open-domain dialogue evaluation has attracted increasing attention.
Trainable evaluation metrics are commonly trained with true positive and
randomly selected negative responses, resulting in a tendency for them to
assign a higher score to the responses that share higher content similarity
with a given context. However, adversarial negative responses possess high
content similarity with the contexts whilst being semantically different.
Therefore, existing evaluation metrics are not robust enough to evaluate such
responses, resulting in low correlations with human judgments. While recent
studies have shown some efficacy in utilizing Large Language Models (LLMs) for
open-domain dialogue evaluation, they still encounter challenges in effectively
handling adversarial negative examples. In this paper, we propose a simple yet
effective framework for open-domain dialogue evaluation, which combines
domain-specific language models (SLMs) with LLMs. The SLMs can explicitly
incorporate Abstract Meaning Representation (AMR) graph information of the
dialogue through a gating mechanism for enhanced semantic representation
learning. The evaluation result of SLMs and AMR graph information are plugged
into the prompt of LLM, for the enhanced in-context learning performance.
Experimental results on open-domain dialogue evaluation tasks demonstrate the
superiority of our method compared to a wide range of state-of-the-art
baselines, especially in discriminating adversarial negative responses. Our
code is available at this https URL

我们提出了一个简单而有效的开放域对话评估框架，通过将领域特定语言模型（SLMs）与 LLMs 结合起来，明确地通过门控机制将对话的抽象意义表示（AMR）图信息融入到 LLM 的提示中，以提高上下文学习性能，并在开放域对话评估任务中的实验结果证明了我们方法在区分对抗性负面回应方面的优越性。