Document-level translation models are usually evaluated using general metrics
such as BLEU, which are not informative about the benefits of context. Current
work on context-aware evaluation, such as contrastive methods, only measure
translation accuracy on words that need context for disambiguation. Such
measures cannot reveal whether the translation model uses the correct
supporting context. We propose to complement accuracy-based evaluation with
measures of context utilization. We find that perturbation-based analysis
(comparing models' performance when provided with correct versus random
context) is an effective measure of overall context utilization. For a
finer-grained phenomenon-specific evaluation, we propose to measure how much
the supporting context contributes to handling context-dependent discourse
phenomena. We show that automatically-annotated supporting context gives
similar conclusions to human-annotated context and can be used as alternative
for cases where human annotations are not available. Finally, we highlight the
importance of using discourse-rich datasets when assessing context utilization.

文件级别的翻译模型通常使用一般性指标（如 BLEU）进行评估，而这些指标对于上下文的益处并不具有信息量。我们的研究提出了补充基于准确性评估的上下文利用度量的方法，并且通过扰动分析发现这是一种有效的全局上下文利用度量。此外，我们还提出了一种细粒度的现象特定评估方法，用于衡量对处理上下文相关的话语现象的支持上下文的贡献。我们的研究结果表明自动注释的支持上下文与人工注释的上下文给出了类似的结论，并可以作为人工注释不可用的情况下的替代方法。最后，我们强调了在评估上下文利用度时使用话语丰富的数据集的重要性。