We explore efficient evaluation metrics for Natural Language Generation
(NLG). To implement efficient metrics, we replace (i) computation-heavy
transformers in metrics such as BERTScore, MoverScore, BARTScore, XMoverScore,
etc. with lighter versions (such as distilled ones) and (ii) cubic inference
time alignment algorithms such as Word Mover Distance with linear and quadratic
approximations. We consider six evaluation metrics (both monolingual and
multilingual), assessed on three different machine translation datasets, and 16
light-weight transformers as replacement. We find, among others, that (a)
TinyBERT shows best quality-efficiency tradeoff for semantic similarity metrics
of the BERTScore family, retaining 97\% quality and being 5x faster at
inference time on average, (b) there is a large difference in speed-ups on CPU
vs. GPU (much higher speed-ups on CPU), and (c) WMD approximations yield no
efficiency gains but lead to a substantial drop in quality on 2 out of 3
datasets we examine.

本文探讨了自然语言生成的有效评估指标，以及通过使用轻量级版本的 Transformer 和线性、二次逼近算法来实现评估指标的高效计算，研究发现，TinyBERT 在语义相似性评估指标方面表现最佳，并且在平均推理时间上比原算法要快 5 倍，但 WMD 近似计算并没有带来效率提升，反而在部分机器翻译数据集上使得质量下降。