Existing evaluation metrics for natural language generation (NLG) tasks face
the challenges on generalization ability and interpretability. Specifically,
most of the well-performed metrics are required to train on evaluation datasets
of specific NLG tasks and evaluation dimensions, which may cause over-fitting
to task-specific datasets. Furthermore, existing metrics only provide an
evaluation score for each dimension without revealing the evidence to interpret
how this score is obtained. To deal with these challenges, we propose a simple
yet effective metric called DecompEval. This metric formulates NLG evaluation
as an instruction-style question answering task and utilizes instruction-tuned
pre-trained language models (PLMs) without training on evaluation datasets,
aiming to enhance the generalization ability. To make the evaluation process
more interpretable, we decompose our devised instruction-style question about
the quality of generated texts into the subquestions that measure the quality
of each sentence. The subquestions with their answers generated by PLMs are
then recomposed as evidence to obtain the evaluation result. Experimental
results show that DecompEval achieves state-of-the-art performance in untrained
metrics for evaluating text summarization and dialogue generation, which also
exhibits strong dimension-level / task-level generalization ability and
interpretability.

本文提出了一种简单而有效的度量方法 DecompEval，该方法在自然语言生成（NLG）评估中使用了指令调整的预训练语言模型，以提高其泛化能力和可解释性。实验结果表明，DecompEval 在未训练的度量标准中取得了最先进的性能，适用于评估文本摘要和对话生成的各种任务和评估维度，并具有较强的泛化能力和可解释性。