Recent studies emphasize the need of document context in human evaluation of
machine translations, but little research has been done on the impact of user
interfaces on annotator productivity and the reliability of assessments. In
this work, we compare human assessment data from the last two WMT evaluation
campaigns collected via two different methods for document-level evaluation.
Our analysis shows that a document-centric approach to evaluation where the
annotator is presented with the entire document context on a screen leads to
higher quality segment and document level assessments. It improves the
correlation between segment and document scores and increases inter-annotator
agreement for document scores but is considerably more time consuming for
annotators.

研究发现，针对机器翻译的人工评估需要考虑文本上下文，然而用户界面对于标注者的生产力和评估可靠性的影响却鲜有研究。本文通过比较两种不同方法获得的人工评估数据，证明了一个以文档为中心的评估方法可以提高数据的质量，但却需要更多的时间投资。