Automatic evaluation is an integral aspect of dialogue system research. The
traditional reference-based NLG metrics are generally found to be unsuitable
for dialogue assessment. Consequently, recent studies have suggested various
unique, reference-free neural metrics that better align with human evaluations.
Notably among them, large language models (LLMs), particularly the
instruction-tuned variants like ChatGPT, are shown to be promising substitutes
for human judges. Yet, existing works on utilizing LLMs for automatic dialogue
evaluation are limited in their scope in terms of the number of meta-evaluation
datasets, mode of evaluation, coverage of LLMs, etc. Hence, it remains
inconclusive how effective these LLMs are. To this end, we conduct a
comprehensive study on the application of LLMs for automatic dialogue
evaluation. Specifically, we analyze the multi-dimensional evaluation
capability of 30 recently emerged LLMs at both turn and dialogue levels, using
a comprehensive set of 12 meta-evaluation datasets. Additionally, we probe the
robustness of the LLMs in handling various adversarial perturbations at both
turn and dialogue levels. Finally, we explore how model-level and
dimension-level ensembles impact the evaluation performance. All resources are
available at this https URL

自动对话评估的研究中，大型语言模型、神经度量指标以及元评估数据集的应用，以及模型层次和维度层次的集成对评估性能的影响进行了全面的研究。