We propose a novel methodology (namely, MuLER) that transforms any
reference-based evaluation metric for text generation, such as machine
translation (MT) into a fine-grained analysis tool.
Given a system and a metric, MuLER quantifies how much the chosen metric
penalizes specific error types (e.g., errors in translating names of
locations). MuLER thus enables a detailed error analysis which can lead to
targeted improvement efforts for specific phenomena.
We perform experiments in both synthetic and naturalistic settings to support
MuLER's validity and showcase its usability in MT evaluation, and other tasks,
such as summarization. Analyzing all submissions to WMT in 2014-2020, we find
consistent trends. For example, nouns and verbs are among the most frequent POS
tags. However, they are among the hardest to translate. Performance on most POS
tags improves with overall system performance, but a few are not thus
correlated (their identity changes from language to language). Preliminary
experiments with summarization reveal similar trends.

MuLER 是一种将文本生成的评估指标转化为细粒度分析工具的新方法，可量化所选度量标准对特定错误类型（例如，位置名称错误）的惩罚程度，并通过分析展示了其在机器翻译等任务中的有效性和可用性。