As automatic metrics become increasingly stronger and widely adopted, the risk of unintentionally "gaming the metric" during model development rises. This issue is caused by metric interference (Mint), i.e., the use of the same or related metrics for both model tuning and evaluation. Mint can misguide practitioners into being overoptimistic about the performance of their systems: as system outputs become a function of the interfering metric, their estimated quality loses correlation with human judgments. In this work, we analyze two common cases of Mint in machine translation-related tasks: filtering of training data, and decoding with quality signals. Importantly, we find that Mint strongly distorts instance-level metric scores, even when metrics are not directly optimized for -- questioning the common strategy of leveraging a different, yet related metric for evaluation that is not used for tuning. To address this problem, we propose MintAdjust, a method for more reliable evaluation under Mint. On the WMT24 MT shared task test set, MintAdjust ranks translations and systems more accurately than state-of-the-art-metrics across a majority of language pairs, especially for high-quality systems. Furthermore, MintAdjust outperforms AutoRank, the ensembling method used by the organizers.

本研究针对在模型开发中使用相同或相关度量导致的度量干扰问题（Mud），提出了MintAdjust方法以提高评估的可靠性。研究发现，度量干扰会严重扭曲实例级别的度量分数，从而导致对系统性能的过度乐观估计。MintAdjust方法在WMT24机器翻译共享任务的测试集上优于现有最先进的度量，特别是在高质量系统中显示出更高的准确性。

在机器翻译中加入巧克力以减轻度量干扰