Ten years ago a single metric, BLEU, governed progress in machine translation
research. For better or worse, there is no such consensus today, and
consequently it is difficult for researchers to develop and retain the kinds of
heuristic intuitions about metric deltas that drove earlier research and
deployment decisions. This paper investigates the "dynamic range" of a number
of modern metrics in an effort to provide a collective understanding of the
meaning of differences in scores both within and among metrics; in other words,
we ask what point difference X in metric Y is required between two systems for
humans to notice? We conduct our evaluation on a new large dataset, ToShip23,
using it to discover deltas at which metrics achieve system-level differences
that are meaningful to humans, which we measure by pairwise system accuracy. We
additionally show that this method of establishing delta-accuracy is more
stable than the standard use of statistical p-values in regards to testset
size. Where data size permits, we also explore the effect of metric deltas and
accuracy across finer-grained features such as translation direction, domain,
and system closeness.

本文研究了现代测量方法的 “动态范围”，旨在提供有关分数差异的集体理解，包括在测量之间以及内部分数之间的意义，即我们问对于人类来说，系统之间需要多大的指标差异 X 才能被注意到。我们在一个新的大型数据集 ToShip23 上进行评估，使用该数据集发现了在评估指标达到对人类有意义的系统级差异时的差值，我们通过成对系统准确性来衡量这种差异。在数据量允许的情况下，我们还探讨了指标差异和准确性对于更细粒度特征的影响，例如翻译方向、领域和系统接近程度。