We propose LLM-Eval, a unified multi-dimensional automatic evaluation method
for open-domain conversations with large language models (LLMs). Existing
evaluation methods often rely on human annotations, ground-truth responses, or
multiple LLM prompts, which can be expensive and time-consuming. To address
these issues, we design a single prompt-based evaluation method that leverages
a unified evaluation schema to cover multiple dimensions of conversation
quality in a single model call. We extensively evaluate the performance of
LLM-Eval on various benchmark datasets, demonstrating its effectiveness,
efficiency, and adaptability compared to state-of-the-art evaluation methods.
Our analysis also highlights the importance of choosing suitable LLMs and
decoding strategies for accurate evaluation results. LLM-Eval offers a
versatile and robust solution for evaluating open-domain conversation systems,
streamlining the evaluation process and providing consistent performance across
diverse scenarios.

本文提出了 LLM-Eval，一种针对使用大型语言模型（LLM）的开放领域对话进行多维自动评估的统一方法。通过设计基于单个提示的评估方法来覆盖会话质量的多个方面，LLM-Eval 可以在单个模型调用中进行。我们对 LLM-Eval 在各种基准数据集上的性能进行了全面评估，表明它相对于最先进的评估方法具有高效性和适应性。同时，该分析还强调了选择适当的 LLM 和解码策略以获得准确评估结果的重要性。LMM-Eval 为评估开放领域对话系统提供了一种多功能且强大的解决方案，可以简化评估过程并在不同场景中提供一致的性能。