Neural metrics for machine translation evaluation, such as COMET, exhibit
significant improvements in their correlation with human judgments, as compared
to traditional metrics based on lexical overlap, such as BLEU. Yet, neural
metrics are, to a great extent, "black boxes" returning a single sentence-level
score without transparency about the decision-making process. In this work, we
develop and compare several neural explainability methods and demonstrate their
effectiveness for interpreting state-of-the-art fine-tuned neural metrics. Our
study reveals that these metrics leverage token-level information that can be
directly attributed to translation errors, as assessed through comparison of
token-level neural saliency maps with Multidimensional Quality Metrics (MQM)
annotations and with synthetically-generated critical translation errors. To
ease future research, we release our code at:
this https URL

本文研究了神经度量与传统度量方法在评估机器翻译中的差异，并开发了多种神经可解释性方法来解释这些度量方法，并证明这些方法能够有效地分析翻译错误，促进未来的研究。

内部故事：迈向更好理解机器翻译神经评估指标

The Inside Story: Towards Better Understanding of Machine Translation  Neural Evaluation Metrics

Recent progress in information retrieval finds that embedding query and
document representation into multi-vector yields a robust bi-encoder retriever
on out-of-distribution datasets. In this paper, we explore whether late
interaction, the simplest form of multi-vector, is also helpful to neural
rerankers that only use the [CLS] vector to compute the similarity score.
Although intuitively, the attention mechanism of rerankers at the previous
layers already gathers the token-level information, we find adding late
interaction still brings an extra 5% improvement in average on
out-of-distribution datasets, with little increase in latency and no
degradation in in-domain effectiveness. Through extensive experiments and
analysis, we show that the finding is consistent across different model sizes
and first-stage retrievers of diverse natures and that the improvement is more
prominent on longer queries.

本文探讨了将后期交互用于神经重排器的有用性，以提高其在分布外数据集中的性能，该方法能带来额外的 5％改进，而无需增加太多的延迟和没有影响其在域内数据上的效果，通过实验和分析，发现这一结论与模型大小和不同类型的第一阶段检索器一致，并且这种改进在较长的查询上更为显著。