Automatic metrics are fundamental for the development and evaluation of
machine translation systems. Judging whether, and to what extent, automatic
metrics concur with the gold standard of human evaluation is not a
straightforward problem. We show that current methods for judging metrics are
highly sensitive to the translations used for assessment, particularly the
presence of outliers, which often leads to falsely confident conclusions about
a metric's efficacy. Finally, we turn to pairwise system ranking, developing a
method for thresholding performance improvement under an automatic metric
against human judgements, which allows quantification of type I versus type II
errors incurred, i.e., insignificant human differences in system quality that
are accepted, and significant human differences that are rejected. Together,
these findings suggest improvements to the protocols for metric evaluation and
system performance evaluation in machine translation.

研究了自动度量在机器翻译系统开发和评估中的问题，发现现有的判断度量方法对于用于评估的翻译非常敏感，特别是存在异常值的情况下，经常会导致关于指标作用的错误结论。研发了一种用于阈值性能提高的算法，可以模拟在与人工判定相比的情况下出现的 2 类误差。这些结果表明，需要改进度量评估和系统性能评估协议。