Evaluation of opinion summaries using conventional reference-based metrics
rarely provides a holistic evaluation and has been shown to have a relatively
low correlation with human judgments. Recent studies suggest using Large
Language Models (LLMs) as reference-free metrics for NLG evaluation, however,
they remain unexplored for opinion summary evaluation. Moreover, limited
opinion summary evaluation datasets inhibit progress. To address this, we
release the SUMMEVAL-OP dataset covering 7 dimensions related to the evaluation
of opinion summaries: fluency, coherence, relevance, faithfulness, aspect
coverage, sentiment consistency, and specificity. We investigate Op-I-Prompt a
dimension-independent prompt, and Op-Prompts, a dimension-dependent set of
prompts for opinion summary evaluation. Experiments indicate that Op-I-Prompt
emerges as a good alternative for evaluating opinion summaries achieving an
average Spearman correlation of 0.70 with humans, outperforming all previous
approaches. To the best of our knowledge, we are the first to investigate LLMs
as evaluators on both closed-source and open-source models in the opinion
summarization domain.

我们使用 SUMMEVAL-OP 数据集评估意见摘要，使用大型语言模型作为参考自由指标，发现 Op-I-Prompt 是评估意见摘要的良好替代方法，与人类判断的平均斯皮尔曼相关性达到 0.70，超过了以前的所有方法。这是我们在意见摘要领域首次探索使用开源和闭源模型的大型语言模型作为评估器。