Word-level Quality Estimation (QE) of Machine Translation (MT) aims to find
out potential translation errors in the translated sentence without reference.
Typically, conventional works on word-level QE are designed to predict the
translation quality in terms of the post-editing effort, where the word labels
("OK" and "BAD") are automatically generated by comparing words between MT
sentences and the post-edited sentences through a Translation Error Rate (TER)
toolkit. While the post-editing effort can be used to measure the translation
quality to some extent, we find it usually conflicts with the human judgement
on whether the word is well or poorly translated. To overcome the limitation,
we first create a golden benchmark dataset, namely \emph{HJQE} (Human Judgement
on Quality Estimation), where the expert translators directly annotate the
poorly translated words on their judgements. Additionally, to further make use
of the parallel corpus, we propose the self-supervised pre-training with two
tag correcting strategies, namely tag refinement strategy and tree-based
annotation strategy, to make the TER-based artificial QE corpus closer to
\emph{HJQE}. We conduct substantial experiments based on the publicly available
WMT En-De and En-Zh corpora. The results not only show our proposed dataset is
more consistent with human judgment but also confirm the effectiveness of the
proposed tag correcting strategies.\footnote{The data can be found at
https://github.com/ZhenYangIACAS/HJQE.}

该论文旨在通过自监督的预训练方法以及标记修正策略，提高机器翻译的质量估计，避免传统质量评价准则的局限性，并通过人类专家的直接评注来构建不需要参考文献的数据集 HJQE 的实验结果证实了我们的方法的有效性。

重新思考基于人工判断的机器翻译单词质量评估

Rethink about the Word-level Quality Estimation for Machine Translation from Human Judgement

Modern neural machine translation (NMT) models have achieved competitive
performance in standard benchmarks such as WMT. However, there still exist
significant issues such as robustness, domain generalization, etc. In this
paper, we study NMT models from the perspective of compositional generalization
by building a benchmark dataset, CoGnition, consisting of 216k clean and
consistent sentence pairs. We quantitatively analyze effects of various factors
using compound translation error rate, then demonstrate that the NMT model
fails badly on compositional generalization, although it performs remarkably
well under traditional metrics.

本论文研究了现代神经机器翻译模型的组成泛化问题，构建了包含 216k 个干净和一致的语句对的基准数据集 CoGnition，并通过复合翻译误差率定量分析了各种因素的影响，然后展示了神经机器翻译模型在组成泛化方面的表现欠佳，尽管在传统指标下表现出色。