Widely used evaluation metrics for text generation either do not work well
with longer texts or fail to evaluate all aspects of text quality. In this
paper, we introduce a new metric called SMART to mitigate such limitations.
Specifically, We treat sentences as basic units of matching instead of tokens,
and use a sentence matching function to soft-match candidate and reference
sentences. Candidate sentences are also compared to sentences in the source
documents to allow grounding (e.g., factuality) evaluation. Our results show
that system-level correlations of our proposed metric with a model-based
matching function outperforms all competing metrics on the SummEval
summarization meta-evaluation dataset, while the same metric with a
string-based matching function is competitive with current model-based metrics.
The latter does not use any neural model, which is useful during model
development phases where resources can be limited and fast evaluation is
required. Finally, we also conducted extensive analyses showing that our
proposed metrics work well with longer summaries and are less biased towards
specific models.

本研究介绍了一种新的文本生成评估指标 SMART，其以句子为匹配的基本单位，通过句子匹配功能软匹配候选句子和参考句子，并与源文档中的句子进行比较以进行评估。实验结果表明，SMART 指标在评估长文本时优于其他评估指标，在无神经网络时再现了基于模型的指标，同时对于长摘要的评估效果良好，且不偏向特定模型。