With the advent of Neural Machine Translation (NMT) systems, the MT output
has reached unprecedented accuracy levels which resulted in the ubiquity of MT
tools on almost all online platforms with multilingual content. However, NMT
systems, like other state-of-the-art AI generative systems, are prone to errors
that are deemed machine hallucinations. The problem with NMT hallucinations is
that they are remarkably \textit{fluent} hallucinations. Since they are trained
to produce grammatically correct utterances, NMT systems are capable of
producing mistranslations that are too fluent to be recognised by both users of
the MT tool, as well as by automatic quality metrics that are used to gauge
their performance. In this paper, we introduce an authentic dataset of machine
translation critical errors to point to the ethical and safety issues involved
in the common use of MT. The dataset comprises mistranslations of Arabic mental
health postings manually annotated with critical error types. We also show how
the commonly used quality metrics do not penalise critical errors and highlight
this as a critical issue that merits further attention from researchers.

介绍了一个关于机器翻译关键错误的真实数据集，指出了常见机器翻译存在的伦理和安全问题以及目前常用质量评估指标无法识别关键错误的重要性。

机器翻译致命错误的网络风险：以阿拉伯语心理健康推文为案例研究

Cyber Risks of Machine Translation Critical Errors : Arabic Mental  Health Tweets as a Case Study

Although neural-based machine translation evaluation metrics, such as COMET
or BLEURT, have achieved strong correlations with human judgements, they are
sometimes unreliable in detecting certain phenomena that can be considered as
critical errors, such as deviations in entities and numbers. In contrast,
traditional evaluation metrics, such as BLEU or chrF, which measure lexical or
character overlap between translation hypotheses and human references, have
lower correlations with human judgements but are sensitive to such deviations.
In this paper, we investigate several ways of combining the two approaches in
order to increase robustness of state-of-the-art evaluation methods to
translations with critical errors. We show that by using additional information
during training, such as sentence-level features and word-level tags, the
trained metrics improve their capability to penalize translations with specific
troublesome phenomena, which leads to gains in correlation with human judgments
and on recent challenge sets on several language pairs.

本研究探讨将传统评估方法如 BLEU 或 chrF 与基于神经网络的机器翻译评估方法如 COMET 或 BLEURT 相结合，通过使用附加的诸如句子级特征和单词级标签等额外信息训练评估指标，提高了最新状态下的机器翻译评估方法的稳健性，从而在几种语言对上提高了与人类判断的相关性和在挑战数据集上获得了更好的表现。