Reliable human evaluation is critical to the development of successful
natural language generation models, but achieving it is notoriously difficult.
Stability is a crucial requirement when ranking systems by quality: consistent
ranking of systems across repeated evaluations is not just desirable, but
essential. Without it, there is no reliable foundation for hill-climbing or
product launch decisions. In this paper, we use machine translation and its
state-of-the-art human evaluation framework, MQM, as a case study to understand
how to set up reliable human evaluations that yield stable conclusions. We
investigate the optimal configurations for item allocation to raters, number of
ratings per item, and score normalization. Our study on two language pairs
provides concrete recommendations for designing replicable human evaluation
studies. We also collect and release the largest publicly available dataset of
multi-segment translations rated by multiple professional translators,
consisting of nearly 140,000 segment annotations across two language pairs.

本文以机器翻译及其人类评估框架 MQM 为案例研究，旨在探讨如何建立可靠的人类评估，以获得稳定的结论，并提供设计可复制的人类评估研究的具体建议，同时发布了由多名专业翻译人员评分的最大公开可用翻译数据集。