The majority of automatic metrics for evaluating NLG systems are reference-based. However, the challenge of collecting human annotation results in a lack of reliable references in numerous application scenarios. Despite recent advancements in reference-free metrics, it has not been well understood when and where they can be used as an alternative to reference-based metrics. In this study, by employing diverse analytical approaches, we comprehensively assess the performance of both metrics across a wide range of NLG tasks, encompassing eight datasets and eight evaluation models. Based on solid experiments, the results show that reference-free metrics exhibit a higher correlation with human judgment and greater sensitivity to deficiencies in language quality. However, their effectiveness varies across tasks and is influenced by the quality of candidate texts. Therefore, it's important to assess the performance of reference-free metrics before applying them to a new task, especially when inputs are in uncommon form or when the answer space is highly variable. Our study can provide insight into the appropriate application of automatic metrics and the impact of metric choice on evaluation performance.

基于广泛的实验，本研究全面评估了参考无关度量与参考度量在各种自然语言生成任务中的性能，并表明参考无关度量与人类判断有较高相关性，并对语言质量的不足更敏感，然而其有效性因任务而异，受候选文本质量影响。因此，在应用参考无关度量到新任务时，特别是在输入形式不常见或答案空间高度变化时，有必要评估其性能。本研究为自动度量的适当应用以及度量选择对评估性能的影响提供了洞察。

在自然语言生成系统评估中，引用是否有必要？何时何地？