We explore the effectiveness of character-level neural machine translation
using Transformer architecture for various levels of language similarity and
size of the training dataset on translation between Czech and Croatian, German,
Hungarian, Slovak, and Spanish. We evaluate the models using automatic MT
metrics and show that translation between similar languages benefits from
character-level input segmentation, while for less related languages,
character-level vanilla Transformer-base often lags behind subword-level
segmentation. We confirm previous findings that it is possible to close the gap
by finetuning the already trained subword-level models to character-level.

我们探讨了基于字符级神经机器翻译的 Transformer 架构在不同语言相似度和训练数据集大小上的效果，针对捷克语、克罗地亚语、德语、匈牙利语、斯洛伐克语和西班牙语之间的翻译进行了评估，使用自动 MT 指标，证明了在相似语言之间，字符级输入分割有益；而对于不相关语言，字符级传统 Transformer 基本落后于子词级分割；我们实验证实了通过微调已经训练好的子词级模型可以弥补这一差距的先前研究结论。

基于字符级神经机器翻译和语言相似度

Character-level NMT and language similarity

Automatic machine translation (MT) metrics are widely used to distinguish the
translation qualities of machine translation systems across relatively large
test sets (system-level evaluation). However, it is unclear if automatic
metrics are reliable at distinguishing good translations from bad translations
at the sentence level (segment-level evaluation). In this paper, we investigate
how useful MT metrics are at detecting the success of a machine translation
component when placed in a larger platform with a downstream task. We evaluate
the segment-level performance of the most widely used MT metrics (chrF, COMET,
BERTScore, etc.) on three downstream cross-lingual tasks (dialogue state
tracking, question answering, and semantic parsing). For each task, we only
have access to a monolingual task-specific model. We calculate the correlation
between the metric's ability to predict a good/bad translation with the
success/failure on the final task for the Translate-Test setup. Our experiments
demonstrate that all metrics exhibit negligible correlation with the extrinsic
evaluation of the downstream outcomes. We also find that the scores provided by
neural metrics are not interpretable mostly because of undefined ranges. Our
analysis suggests that future MT metrics be designed to produce error labels
rather than scores to facilitate extrinsic evaluation.

本文研究了自动机器翻译度量在句子级别（段落级别评估）中区分好的翻译和坏的翻译的可靠性，并研究了在较大平台中放置机器翻译组件的成功率检测中 MT 度量的有用性。我们在三个下游跨语言任务（对话状态跟踪，问题回答和语义分析）上评估了最广泛使用的 MT 度量（chrF，COMET，BERTScore 等）的段落级别性能。我们的实验表明，所有的度量标准与下游结果的内在评估显示出微不足道的相关性。我们还发现，神经度量提供的分数大多数不是可解释的，因为其值域未定义。我们的分析表明，将来的 MT 指标应该被设计成产生错误标签而不是得分，以便于外在评估。

机器翻译指标外部评估

Extrinsic Evaluation of Machine Translation Metrics

As machine translation (MT) metrics improve their correlation with human
judgement every year, it is crucial to understand the limitations of such
metrics at the segment level. Specifically, it is important to investigate
metric behaviour when facing accuracy errors in MT because these can have
dangerous consequences in certain contexts (e.g., legal, medical). We curate
ACES, a translation accuracy challenge set, consisting of 68 phenomena ranging
from simple perturbations at the word/character level to more complex errors
based on discourse and real-world knowledge. We use ACES to evaluate a wide
range of MT metrics including the submissions to the WMT 2022 metrics shared
task and perform several analyses leading to general recommendations for metric
developers. We recommend: a) combining metrics with different strengths, b)
developing metrics that give more weight to the source and less to
surface-level overlap with the reference and c) explicitly modelling additional
language-specific information beyond what is available via multilingual
embeddings.

使用 ACES 挑战集评估各种机器翻译评估指标，分析评估结果并提出了开发者的一些共同建议，包括组合不同的度量，发展更多强调从源头到引用的度量，并明确建模多语言表达的更多信息。