We explore efficient evaluation metrics for Natural Language Generation (NLG). To implement efficient metrics, we replace (i) computation-heavy transformers in metrics such as BERTScore, MoverScore, BARTScore, XMoverScore, etc. with lighter versions (such as distilled ones) and (ii) cubic inference time alignment algorithms such as Word Mover Distance with linear and quadratic approximations. We consider six evaluation metrics (both monolingual and multilingual), assessed on three different machine translation datasets, and 16 light-weight transformers as replacement. We find, among others, that (a) TinyBERT shows best quality-efficiency tradeoff for semantic similarity metrics of the BERTScore family, retaining 97\% quality and being 5x faster at inference time on average, (b) there is a large difference in speed-ups on CPU vs. GPU (much higher speed-ups on CPU), and (c) WMD approximations yield no efficiency gains but lead to a substantial drop in quality on 2 out of 3 datasets we examine.

本文探讨了自然语言生成的有效评估指标，以及通过使用轻量级版本的Transformer和线性、二次逼近算法来实现评估指标的高效计算，研究发现，TinyBERT在语义相似性评估指标方面表现最佳，并且在平均推理时间上比原算法要快5倍，但WMD近似计算并没有带来效率提升，反而在部分机器翻译数据集上使得质量下降。

能否更简单地做到这一点？自然语言生成的简单、高效、高质量评估指标