We benchmark the performance of segmentlevel metrics submitted to WMT 2023 using the ACES Challenge Set (Amrhein et al., 2022). The challenge set consists of 36K examples representing challenges from 68 phenomena and covering 146 language pairs. The phenomena range from simple perturbations at the word/character level to more complex errors based on discourse and real-world knowledge. For each metric, we provide a detailed profile of performance over a range of error categories as well as an overall ACES-Score for quick comparison. We also measure the incremental performance of the metrics submitted to both WMT 2023 and 2022. We find that 1) there is no clear winner among the metrics submitted to WMT 2023, and 2) performance change between the 2023 and 2022 versions of the metrics is highly variable. Our recommendations are similar to those from WMT 2022. Metric developers should focus on: building ensembles of metrics from different design families, developing metrics that pay more attention to the source and rely less on surface-level overlap, and carefully determining the influence of multilingual embeddings on MT evaluation.

通过使用ACES Challenge Set作为评估标准，我们对提交给WMT 2023的分段级别指标的性能进行基准测试，发现没有明显的优胜者，并且在2023和2022版本之间的性能变化高度不确定。我们建议指标开发者着重构建来自不同设计家族的指标集合，开发更加注重源信息并减少表面重叠依赖的指标，并仔细确定多语言嵌入对机器翻译评估的影响。

WMT 2023年ACES：翻译准确度挑战集