In this study, we analyze NLG automatic metrics based on whether human
evaluation aspect is used as context or objective to compute the metrics: (i)
Task-agnostic and (ii) Human-aligned. Task-agnostic metrics, such as
Perplexity, BLEU, BERTScore, are cost-effective and highly adaptable to diverse
NLG tasks, yet they have a weak correlation with human. Human-aligned metrics
(CTC, CtrlEval, UniEval) improves correlation level by incorporating desirable
human-like qualities as training objective. However, their effectiveness at
discerning system-level performance and quality of system outputs remains
unclear.
We present metric preference checklist as a framework to assess the
discriminative power of automatic metrics in three NLG tasks: Text
Summarization, Dialogue Response Generation, and Controlled Generation. We show
that multi-aspect human-aligned metric (UniEval) is not necessarily dominant
over single-aspect human-aligned metrics (CTC, CtrlEval) and task-agnostic
metrics (BLEU, BERTScore), particularly when a disagreement between human
evaluation aspects is present. We also show particular use cases in which
automatic metrics provide a better guidance than human on discriminating
system-level performance. Our proposed framework provides access: (i) for
verifying whether automatic metrics are faithful to human preference,
regardless their correlation level to human; and (ii) for scrutinizing the
strengths and limitations of NLG systems, which are often obscured by a
standard averaging method of evaluation scores.

该研究分析了基于人类评估方面作为上下文或目标来计算 NLG 自动度量的自动度量，并提出了度量偏好清单作为评估自动度量在三个 NLG 任务中的区分能力的框架。研究显示，多方面的人性化度量并不一定比单方面的人性化度量和任务不可知度量更为优越，并且自动度量在一些情况下提供了比人类更好的指导。该框架提供了验证自动度量是否忠实于人类偏好的访问，以及审查 NLG 系统的优势和局限性的能力。